From unlocking a smartphone to searching millions of images, facial recognition turns the geometry and visual characteristics of a human face into mathematical data. But how does a computer actually recognise a person — and what happens when it gets the answer wrong?

By Open Chronicle Explained

A face is one of the most familiar things in the human world.

We recognise friends across crowded rooms. We identify relatives in old photographs. We notice when someone has changed their hairstyle, grown a beard or aged.

For computers, however, a face begins as something very different.

Pixels.

A facial-recognition system must transform those pixels into information that can be processed mathematically. It must locate a face, extract characteristics from it, create a numerical representation and compare that representation with another one — or potentially with millions of them.

The basic idea sounds simple.

The engineering behind it is not.

Modern facial recognition combines computer vision, machine learning, cameras, databases and statistical decision-making. The technology can authenticate the owner of a smartphone, help verify travellers against passport photographs or search collections of facial images for possible matches.

Those applications may all involve faces, but they do not necessarily present the same technical problem — or the same consequences when the system makes a mistake.

Understanding facial recognition therefore begins with an important distinction:

recognising a face is not the same thing as understanding a face.

The machine is trying to determine whether two mathematical representations are sufficiently similar to indicate the same identity.

Here is how that process works.

1. What Is Facial Recognition?

Facial recognition is a biometric technology designed to establish or verify identity using characteristics of a person’s face.

The word biometric is important.

Passwords depend on something you know.

Security tokens depend on something you possess.

Biometric systems attempt to use something associated with your physical or behavioural characteristics — such as a fingerprint, iris, voice or face.

But facial recognition should also be distinguished from several related technologies.

Face detection asks:

Is there a face in this image?

Face analysis might attempt to estimate characteristics from a face.

Facial recognition asks a different question:

Does this face correspond to a particular identity?

A camera detecting that three people are standing in a photograph does not necessarily know who those people are.

Identification requires additional processing.

And the first step is finding the face.

2. First, the Computer Has to Find a Face

Imagine a photograph containing a person standing beside a car, a building and a tree.

To a computer, the original image is essentially an array of pixel values.

Before recognition can begin, the system must determine which part of that image contains a face.

This is face detection.

Modern computer-vision systems can identify patterns associated with faces and locate them within photographs or video frames.

The result might effectively look like a box drawn around the face:

IMAGE → FACE DETECTED → FACE REGION EXTRACTED

Once the face has been located, the system can concentrate on that region instead of analysing the entire scene.

But simply cropping out the face is not enough.

Faces appear under very different conditions.

A person may be:

• looking slightly sideways,
• standing farther from the camera,
• smiling,
• wearing glasses,
• photographed under poor lighting,
• partially obscured,
• or captured by a low-resolution camera.

The system therefore usually needs to prepare the image before attempting a comparison.

3. The Face Must Be Normalised

Imagine comparing two passport photographs.

In one, the face occupies almost the entire frame.

In another, the person is slightly farther away and their head is tilted.

Directly comparing the pixels would be unreliable.

Facial-recognition systems therefore commonly perform forms of alignment and normalisation.

The software may identify important facial reference points — often called landmarks — associated with areas such as the eyes, nose, mouth and outline of the face.

These landmarks help the system align the facial image into a more consistent orientation.

Conceptually, the process looks like this:

Detect face

↓

Locate facial landmarks

↓

Correct orientation and scale

↓

Prepare standardised facial image

This makes comparisons more consistent.

But modern facial recognition does not normally stop at measuring distances between a handful of facial landmarks.

Machine learning takes the process much further.

4. How Does a Machine Turn a Face Into Numbers?

This is the central idea behind modern facial recognition.

A neural network analyses the face and transforms information from the image into a mathematical representation.

This representation is often called a:

facial template

or, in many machine-learning systems, an:

embedding

An embedding can be thought of as a position in a mathematical space with many dimensions.

Instead of storing the concept:

brown eyes, certain nose shape, particular jawline

the model generates a collection of numerical values representing identity-related patterns it has learned to extract from faces.

A simplified example might look like:

Face

↓

[0.18, -0.74, 0.33, 0.91, -0.12 ...]

A real representation can contain many more dimensions.

The individual numbers are not usually meaningful to a human reader.

Their power comes from their relationship to one another.

Faces belonging to the same person should ideally produce representations that lie relatively close together in this mathematical space.

Different people should produce representations that are farther apart.

This changes the recognition problem.

The computer no longer has to ask:

Do these two photographs look identical?

Instead, it can ask:

How similar are these two mathematical representations?

5. What Is a Facial Embedding?

Imagine an enormous map.

Every face processed by the recognition model receives a location somewhere on that map.

Photographs of the same person taken under somewhat different conditions should ideally appear near one another.

Photographs of different people should appear farther apart.

The real mathematical space has far more dimensions than we can conveniently visualise, but the principle can be simplified:

PHOTO A

↓

Neural network

↓

FACIAL EMBEDDING A

versus

PHOTO B

↓

Neural network

↓

FACIAL EMBEDDING B

The system then measures the similarity or distance between them.

This is one reason modern facial recognition can cope with differences between two photographs.

The goal is not to demand pixel-for-pixel identity.

The goal is to extract features useful for recognising identity despite reasonable variation in appearance.

6. How Are Two Faces Compared?

Suppose you enrol your face in a biometric authentication system.

The system creates a reference representation of your face.

Later, another facial image is captured.

A new representation is generated.

The two representations are compared.

The result can be expressed as a similarity score.

Conceptually:

ENROLLED FACE

↓

Facial template A

↘

SIMILARITY

↗

Facial template B

↑

NEW FACE

The higher the similarity, the stronger the evidence that the images represent the same person — although the precise interpretation depends on the algorithm.

But the system still needs to make a decision.

For that it needs a threshold.

7. What Is a Similarity Threshold?

Facial recognition is generally not based on the computer discovering that two representations are perfectly identical.

Instead, the system determines whether the similarity between them is sufficiently high.

Imagine a hypothetical scale:

0 ——————————————— 1

Different person Very similar

A system might establish a decision threshold somewhere along that scale.

If the score is above the threshold, the system may accept the comparison as a match.

If it is below the threshold, it may reject it.

This creates an important trade-off.

Make the threshold extremely strict and legitimate users may be rejected more often.

Make it too permissive and different people may be incorrectly matched.

The appropriate threshold therefore depends heavily on what the system is being used for.

Unlocking a personal device is not the same problem as searching a database containing millions of identities.

8. Verification vs Identification: 1:1 and 1

This is one of the most important distinctions in facial recognition.

There are two fundamental types of comparison.

One-to-One Verification

The system asks:

Is this person who they claim to be?

This is written as:

1:1

A captured face is compared with one enrolled identity.

For example:

FACE PRESENTED

↓

Compare with

↓

JOSÉ’S ENROLLED FACE

↓

MATCH / NO MATCH

Smartphone authentication is a familiar example of this general type of problem.

The search space is tiny.

The system already knows which identity it is testing.

One-to-Many Identification

Now imagine a different question:

Who is this person?

The system may compare one facial representation against a database containing many identities.

This is:

1

Conceptually:

UNKNOWN FACE

↓

Compare against

↓

Person 1
Person 2
Person 3
Person 4
…
Person N

↓

POSSIBLE CANDIDATES

This is fundamentally different.

The larger the search environment and the more consequential the application, the more important error rates, thresholds, image quality and operational safeguards become.

A candidate returned by a facial-recognition system should not automatically be interpreted as proof of identity.

9. What Happens When You Unlock Your Phone With Your Face?

Smartphone authentication provides a useful example of facial verification.

Different manufacturers implement biometric authentication differently, but Apple’s Face ID demonstrates how sophisticated such a system can become.

Apple says its TrueDepth camera projects and analyses thousands of invisible infrared points to construct a depth map of the user’s face while also capturing an infrared image.

A protected neural-processing system then transforms this information into a mathematical representation and compares it with enrolled facial data.

The comparison happens within Apple’s Secure Enclave.

The significant point is that the system is not simply storing an ordinary portrait photograph and asking whether the new photograph looks similar.

It is using specialised sensors, depth information, machine learning and a protected biometric representation.

Apple also incorporates mechanisms designed to determine attention and resist attempts to fool the system with photographs or masks.

This demonstrates another important point:

facial recognition is not one single technology.

Different applications can use different sensors, models, security architectures and databases.

10. How Can a System Search Millions of Faces?

Identification systems face a different challenge.

Suppose investigators have an image of an unknown person.

The system creates a facial representation from that image.

It can then compare that representation against representations associated with a database.

At the simplest conceptual level:

UNKNOWN FACE

↓

Create embedding

↓

Search facial database

↓

Calculate similarities

↓

Rank possible candidates

↓

Return potential matches

Searching enormous databases efficiently requires sophisticated indexing and search techniques, but the underlying concept remains similar.

The system is looking for mathematical representations that most closely resemble the query representation.

Crucially, the output may be a candidate list, not an infallible declaration of identity.

That distinction becomes especially important in high-consequence settings.

11. Why Isn’t Facial Recognition 100% Accurate?

Because the real world is messy.

A facial-recognition system may encounter:

• poor lighting,
• shadows,
• low resolution,
• motion blur,
• unusual camera angles,
• ageing,
• facial hair,
• makeup,
• glasses,
• masks,
• facial expressions,
• partial obstruction,
• compression artefacts,
• and differences between cameras.

Even excellent algorithms cannot recover information that a photograph never captured.

Image quality therefore matters enormously.

The algorithm itself matters too.

Independent evaluations by the US National Institute of Standards and Technology have repeatedly demonstrated substantial differences in performance between facial-recognition algorithms.

It is therefore misleading to ask simply:

How accurate is facial recognition?

A better question is:

Which algorithm, operating on which images, under which conditions, at which threshold, for which task?

12. False Positives and False Negatives

There are two fundamental ways a recognition system can make a matching error.

False Positive

The system incorrectly associates two different people.

PERSON A ≠ PERSON B

but the system returns:

MATCH

False Negative

The system fails to associate two images of the same person.

PERSON A = PERSON A

but the system returns:

NO MATCH

Neither error exists in isolation from the system’s threshold and application.

Consider smartphone authentication.

A false negative might mean the legitimate owner has to try again or enter a passcode.

In a large-scale identification system, however, an incorrect candidate can have much more serious consequences if humans or institutions treat the algorithmic result as definitive evidence.

Context changes the meaning of an error.

13. Does Facial Recognition Work Equally Well for Everyone?

Not necessarily.

This question has been extensively studied, including through NIST’s facial-recognition evaluations.

NIST has found demographic differentials in many algorithms it has examined, but those differences vary considerably among algorithms.

They can also be affected by image quality and other operational conditions.

For example, NIST notes that inadequate lighting can contribute to false-negative differences if darker skin is underexposed or lighter skin is overexposed.

False-positive demographic differentials can persist even with good image quality and may be related, among other factors, to the data used to train a system.

The important conclusion is therefore more nuanced than saying simply that “facial recognition is biased.”

Different algorithms behave differently.

Performance must be measured.

Datasets matter.

Camera conditions matter.

Thresholds matter.

And demographic performance must be evaluated rather than assumed.

14. Can Masks, Ageing and Camera Angles Confuse It?

Yes — although modern systems have become much better at handling variation.

Consider what can change between two photographs of the same person.

Today

Clean-shaven
Glasses
Bright daylight
Front-facing camera

Five years later

Beard
No glasses
Indoor lighting
Slightly angled camera

A robust recognition system should still extract enough identity-related information to associate the images.

But every change potentially makes the task harder.

Ageing can gradually alter facial appearance.

Extreme pose can hide important regions.

Low resolution removes detail.

Masks cover parts of the face.

Poor illumination changes the visual information available to the model.

Modern systems attempt to learn representations that remain useful despite these changes, but no algorithm can eliminate every source of uncertainty.

15. Can Facial Recognition Be Fooled?

Security engineers refer to attempts to impersonate another person to a biometric system as forms of presentation attack or spoofing.

A basic facial-recognition system relying only on an ordinary image could face obvious problems if it cannot distinguish a live person from a photograph or screen.

More sophisticated authentication systems therefore incorporate additional protections.

These can include:

• depth sensing,
• infrared information,
• attention detection,
• liveness mechanisms,
• anti-spoofing neural networks,
• or combinations of several signals.

Apple, for example, says Face ID uses depth information unavailable in ordinary printed or two-dimensional photographs and employs neural networks intended to resist spoofing.

But anti-spoofing is itself an engineering contest.

As recognition technology develops, attempts to deceive biometric systems develop too.

That means security cannot depend solely on the recognition algorithm.

16. Where Is Facial Recognition Used?

Facial recognition can appear in many different environments.

Among them:

Personal devices

Biometric authentication for smartphones and computers.

Border processing

Comparison of travellers with identity-document photographs.

Airports

Identity verification during parts of passenger processing.

Access control

Entry to buildings or restricted environments.

Financial services

Identity verification in some onboarding and authentication processes.

Photograph organisation

Grouping or identifying people in personal image collections, depending on the service.

Security and law enforcement

Searching facial imagery against authorised databases in jurisdictions and circumstances where such use is permitted.

These applications should not automatically be treated as equivalent.

A voluntary 1:1 authentication performed locally on someone’s phone presents very different privacy, accuracy and governance questions from remote identification of people moving through a public space.

17. Why Is Facial Recognition Different From Ordinary CCTV?

Traditional CCTV primarily captures images.

Facial recognition adds another layer:

identity processing.

Consider the difference.

Conventional camera

Person enters station

↓

Camera records person

↓

Video stored or monitored

Now add facial recognition.

Camera + facial recognition

Person enters station

↓

Face detected

↓

Facial representation generated

↓

Representation compared with database

↓

Potential identity candidate generated

This transforms the system.

The question is no longer simply:

What happened here?

It can become:

Who is this person?

And if cameras and identification systems operate at scale, another possibility emerges:

Where has this person appeared?

That is why remote biometric identification has generated substantial legal and civil-liberties debate.

18. What Happens to Your Biometric Data?

This is one of the most important questions surrounding the technology.

A password can be changed.

Your face cannot easily be replaced.

That makes the security architecture around biometric information particularly important.

But again, implementations differ.

Apple says Face ID mathematical representations are encrypted, protected by the Secure Enclave, remain on the device and are not backed up to iCloud.

Other facial-recognition systems may use centralised databases.

That changes the risk profile.

Important questions include:

Where is the biometric representation stored?

Who controls it?

How long is it retained?

Can it be linked with other information?

Who can search it?

Can the individual challenge an incorrect match?

Can the biometric record be deleted?

The recognition algorithm is therefore only one component of the larger system.

Database architecture and governance can matter just as much.

19. Why Has Facial Recognition Become Controversial?

Because facial recognition sits at the intersection of several powerful capabilities.

AI

Biometrics

Cameras

Identity databases

Large-scale computing

Individually, each technology can serve useful purposes.

Combined, they can create powerful identification infrastructure.

Supporters of particular applications point to convenience, security, fraud prevention, authentication and investigative value.

Critics and civil-liberties organisations raise concerns about privacy, surveillance, consent, demographic performance, misuse, data retention and the possibility of identifying people at scale without their active participation.

The appropriate balance depends partly on the application.

Unlocking a device someone voluntarily enrolled is different from identifying strangers walking through a public square.

That distinction increasingly appears in regulation.

20. How Is Europe Regulating Facial Recognition?

The European Union’s AI Act places particular restrictions around biometric identification.

Under the framework, real-time remote biometric identification in publicly accessible spaces for law-enforcement purposes is prohibited in general, while narrowly defined exceptions can be permitted under specified conditions.

Those exceptions concern circumstances such as targeted searches for certain victims or missing persons, specified serious crimes and prevention of serious threats, and are subject to safeguards including necessity, proportionality and authorisation requirements.

The European framework also distinguishes real-time identification from some uses involving previously collected material.

The distinction illustrates an important principle:

“facial recognition” is too broad a term to describe the risk of a system by itself.

Who uses it, where it operates, what database it searches, whether people know it is operating and what happens after a match can be just as important as the algorithm.

21. Could AI Make Facial Recognition Much More Powerful?

AI has already transformed facial recognition.

Deep neural networks have dramatically improved the ability of algorithms to extract useful identity representations from images.

But several broader technological developments could increase the capability of biometric systems further.

Better cameras produce better imagery.

More efficient AI allows processing to happen faster.

Edge computing can move analysis closer to cameras.

Large databases increase the number of identities that can potentially be searched.

Systems can also combine multiple signals.

Imagine an identity system using not only a face but additional authorised biometric or contextual information.

The technical possibility does not automatically determine whether such a system should be deployed.

That becomes a question of law, policy, proportionality and social choice.

The more powerful identification becomes, the more important governance becomes alongside engineering.

22. Where Does Human Oversight Fit In?

A similarity score is not the same thing as certainty.

This becomes particularly important in high-consequence identification.

Suppose an algorithm searches a database and returns several possible candidates.

The system has not necessarily “discovered the person.”

It has produced information that must be interpreted.

A responsible workflow may therefore look more like:

IMAGE

↓

FACIAL RECOGNITION SEARCH

↓

CANDIDATE

↓

HUMAN REVIEW

↓

ADDITIONAL EVIDENCE

↓

DECISION

rather than:

FACIAL RECOGNITION = IDENTITY PROVEN

Human oversight does not magically eliminate error.

Humans can also make mistakes or place too much confidence in algorithmic suggestions.

But separating an algorithmic candidate from a consequential decision is an important conceptual safeguard.

23. What Does the Future of Facial Recognition Look Like?

Facial recognition is likely to become less visible even as the underlying technology becomes more capable.

The extraordinary computational process can happen in fractions of a second.

A person looks at a device.

The device unlocks.

Behind that seemingly simple event may be:

Face detection

↓

Image capture

↓

Alignment

↓

Neural processing

↓

Mathematical representation

↓

Similarity comparison

↓

Threshold decision

↓

Authentication

The same underlying concepts can also operate at far greater scale.

That is what makes facial recognition simultaneously useful and consequential.

The technology can make identity verification nearly frictionless.

But the ability to transform a human face into searchable data changes what cameras and databases can do.

The central challenge is therefore no longer merely whether machines can recognise faces.

Increasingly, it is determining when they should.

What?

Facial recognition is a biometric technology that uses computer vision and machine learning to compare mathematical representations derived from human faces.

It can be used for verification, where one face is compared with one enrolled identity, or identification, where one face may be searched against many identities.

How?

A typical system follows a chain resembling:

Camera

↓

Face Detection

↓

Face Alignment

↓

Feature Extraction

↓

Facial Embedding

↓

Comparison

↓

Similarity Score

↓

Decision Threshold

↓

Match / No Match — or Candidate List

The exact implementation varies substantially between systems.

Why?

Because the human face offers a convenient biometric signal that can often be captured without physical contact.

That makes facial recognition useful for authentication and identity verification.

But it also makes facial recognition unusually powerful.

Unlike a password, a face is routinely visible in public.

A technology capable of converting that face into searchable identity data therefore creates questions extending far beyond computer science.

Those questions concern privacy, security, fairness, consent, surveillance, regulation and the relationship between individuals and institutions.

Your Face as Data

For thousands of years, recognising another human face was primarily a biological ability.

Humans did it instinctively.

Computers changed the scale of the problem.

A machine does not need to remember a face as humans do.

It can convert facial information into mathematics.

That representation can be compared in milliseconds.

One face can be checked against another.

One face can potentially be searched against millions.

And cameras can capture faces continuously.

This is the fundamental transformation created by facial recognition.

The technology is not simply teaching computers to see us.

It is making aspects of human identity computable.

That can make everyday technology extraordinarily convenient.

It can help establish identity in situations where doing so is legitimate and useful.

It can also create powerful systems of identification whose consequences depend on far more than their technical accuracy.

Understanding facial recognition therefore requires understanding both sides of the equation:

how the machine recognises a face — and what society chooses to do with the answer.

Open Chronicle Explained

What? How? Why?

Understanding the systems behind the headlines.

Explore Open Chronicle Explained →