We know that something is up with the way the universe is expanding - there’s some kind of anti-gravitational effect that’s causing the expansion to accelerate.

We don’t know what it is - just that it competes against the inward-pulling effect of gravity.

And it’s winning - it looks like the universe will expand forever, at an ever-increasing rate.

We call this mysterious influence dark energy, but while we’ve talked a lot about how it behaves, we’ve never really explored what it is.

So, what is dark energy, really?

The mainstream physical explanation for dark energy is that the vacuum of space has a constant energy density.

Empty space buzzes with random activity that we sometimes describe as virtual particles popping into and out of existence, driving accelerated expansion.

But there are enough problems with this that we need to explore other options for what dark energy might be, and how it might behave.

We’ve already mentioned one of the alternative behaviors: the case where dark energy gets more intense over time rather than having a constant density expected of a vacuum energy.

That would cause the universe to eventually tear itself apart on a subatomic scale in the so-called big rip.

But we never talked about what might cause dark energy to behave this way.

The default model for dark energy is that it can be described with a so-called cosmological constant.

This is just a static number that you can add to Einstein’s equations of general relativity to represent the fabric of space having a constant, non-zero energy density.

If the fabric of space has energy - dark energy - then the expansion of the universe creates the stuff.

And it’s this process that actually accelerates the expansion, which in turn accelerates the creation of dark energy, and so on, ultimately leading to exponential acceleration.

Exactly why the creation of dark energy accelerates expansion requires a dive into the hairy math of general relativity - and we tried that in a couple of previous episodes.

For today, I need to ask you to accept that the anti-gravitational effect of dark energy is due to it having negative pressure.

That’s counter-intuitive because negative pressure is an inward pulling pressure.

But in an expanding universe negative pressure actually energizes the expansion.

Another slightly counter-intuitive thing about dark energy is that, on top of the anti-gravitational effect of its negative pressure, it also produces regular attractive gravity due to its positive energy density, just like regular matter and energy do.

But the effect of its negative pressure is more powerful, so the net effect is antigravitational.

We can describe the “power” of any dark energy candidate by taking the ratio of pressure to density.

This gives us the equation of state of dark energy, with the ratio equated to the equation of state parameter, omega.

The value of omega tells you almost everything about what your dark energy candidate will do to the universe.

For dark energy, omega is negative due to the negative pressure on the top of the fraction and the positive energy density on the bottom.

Standard “cosmological constant” dark energy has omega -1.

This is just what you get when you say that the vacuum has a constant energy density.

If we assume this is true we can figure out what energy density would be needed to cause the acceleration we observe.

That number is around 5x10^-10 Joules/m^3.

To give you a sense of the smallness of this, if you could take all the dark energy in a space the size of the earth and convert it to mass by E=mc^2, you’d only get about a grain of sand’s worth of matter.

That’s such a tiny amount that it doesn’t have any real effect in places where there’s even a smattering of regular matter - like inside our solar system, or even inside our galaxy.

But if you include the vast voids between the galaxies, the extremely diffuse dark energy adds up and ultimately this becomes the dominant form of energy in our universe.

So its antigravitational effect not only overcomes its own positive gravity, but also the positive gravity of all other matter.

And it’s only getting started.

As the universe gets bigger and galaxies get further and further apart, their gravitational connection dilutes away, while dark energy just keeps pushing and pushing and pushing.

If we can measure the energy density, can’t we also measure omega, the equation of state?

Well, yeah, and it’s pretty much … -1, the default option.

Our most precise measurement comes from Planck satellite data, which mapped the cosmic microwave background.

The Planck team measured w = −1.028 ± 0.032.

So, while this is perfectly consistent with omega=-1, there’s still some wiggle room for omega not quite equal to -1.

But if evidence is pointing to omega=-1, and we have a plausible physical explanation for omega=-1, why should we spend youtube episodes and embarrassingly meager funding of theoretical physicists exploring other options?

Well, I already mentioned that there are problems with the explanation where dark energy is due to quantum fluctuations.

For example, it’s actually very difficult to get the vacuum energy to be very close to zero but not quite equal to zero.

A naive calculation for the energy of the vacuum gives you a number something like 120 orders of magnitude larger than the measured energy density of dark energy.

You can reduce that number if the quantum fields sort of cancel each other out.

A perfectly symmetric canceling could get you down to a vacuum energy of zero, but it’s really really hard to cancel such a large number down to nearly zero but not quite zero.

This is the cosmological constant problem, and we’ve discussed it previously.

A second reason is the so-called Hubble tension.When we measure the Hubble constant - the current rate of expansion of the universe, based on supernova explosions over the past several billion years, we get one number for that expansion rate.

But if we calculate the current expansion rate based on observations of the very early universe we get a different number.

The latter is from Planck satellite measurements of the cosmic microwave background.

The current thinking is that one of these measurements has a very subtle error.

But it could also be that our assumed cosmology is wrong.

In order to calculate the Hubble constant from the early universe measurements, we also need to assume an equation of state to see how the expansion rate should have evolved into the late universe.

The Planck measurement assumes a constant dark energy and an unchanging omega of -1.

But if dark energy has changed over time, both teams may have got the right answer and the discrepancy between their results could actually tell us something about what dark energy really is.

And there’s a final reason to look for a mechanism for accelerating expansion due being some other than quantum fluctuations.

It’s because we know there must be one to explain cosmic inflation.

This was a period of extreme exponential expansion that likely occurred during the big bang.

That expansion must have been due to one or more quantum fields being in a highly energetic state, rather than all quantum fields fluctuating a teensy bit above their energy minima.

So if a specific field was responsible for inflation, couldn’t one also be responsible for dark energy?

The answer is yes, one could - and if that’s true then a lot of problems with cosmological constant dark energy could be solved.

There are a few options for dark energy as a new quantum field.

Perhaps the most prominent is quintessence, proposed by Robert Caldwell, Rahul Dave and Paul Steinhardt in 1998, the same year as the acceleration of the universe was discovered.

The name quintessence comes from Medieval Latin, and means ‘fifth essence.’ The alchemists thought everything in the world was made of the four Aristotlean elements, water, earth, fire, and air, while another fifth element called quintessence filled the celestial spheres beyond the earth.

It’s an apt because it's field fills all of space and also it can be thought of as another force on top of the commonly known 4 fundamental forces.

Alternatively, it can be thought of as a fifth energetic component of the universe on top of baryons, dark matter, neutrinos, and photons.

The quintessence field would have to be a scalar field, like the Higgs field.

So, it would take on a simple numerical value - a field strength - everywhere in space.

The equation of state depends on this field strength and the kinetic energy of the particles of the field.

The field strength can also change over time AND over space, so omega can change, and with it the behavior of dark energy.

This dynamical nature of quintessence is what makes it so powerful.

For example, if the strength of dark energy has changed since the early universe then the Hubble tension could be explained.

Quintessence can also help solve some of the uncomfortable coincidences that seem necessary with a cosmological constant dark energy.

Currently around 70% of the energy in the universe is dark energy with the remaining 30% mostly matter, including dark matter.

That doesn’t sound very close, but it actually is.

As the universe expands, matter dilutes away while most versions of dark energy stay constant or relatively constant, or even increase in some models.

The universe will spend the vast, vast majority of both its past and future history with a huge difference between the densities of dark energy and matter.

So why do we happen to live in a period of the universe where these are within a factor of a few of each other, at exactly the time when the acceleration begins?

This could be a crazy coincidence, or perhaps there are anthropic arguments for this - for example we could only exist in such a time.

But quintessence actually gives us another explanation.

The quintessence field could be coupled to the quantum fields responsible for radiation and matter, and its behavior could be connected to the density of the universe.

For example, in so-called k-essence models, the equation of state is connected to the density of matter.

And it only becomes dark-energy-like when matter starts to thin out.

That provides a natural explanation for why dark energy kicked in at around the same time as stars and planets were able to form.

This same “tracker” behavior could also help solve the cosmological constant problem.

If quintessence shifts to match the matter fields, it could potentially cancel out their predicted extremely high vacuum energy in a natural way, so that dark energy becomes the tiny residual from this canceling.

OK, so quintessence seems promising.

Let’s look at how it actually behaves.

As I mentioned, the behavior of dark energy is driven by its equation of state.

Any value of omega less than -⅓ means accelerating expansion, and omega -1 is a constant energy density.

Quintessence is often used to refer to any omega between -⅓ and -1, while values less than -1 are usually called phantom energy.

However the labels are a little bit ambiguous, and all of these could result from a new scalar quantum field.

If omega is between -⅓ and -1 then the outward-pushing negative pressure still dominates over the inward pull of regular gravity, so expansion still accelerates - but that expansion is restricted to regions outside galaxies.

The Milky Way survives a quintessence-dominated universe.

That’s not true if omega is less than -1 - that’s the big rip, in which all points in space eventually become infinitely far apart.

Fortunately, that seems unlikely.

The only way to get omega less than -1 is for the kinetic energy of the field to be negative.

These sorts of negative energy scenarios break the rules in general relativity in the same way that time travel does.

So, if the big rip is possible, so are time machines, and we escape the end of the universe every time it comes up.

Quintessence is an extremely flexible theoretical mechanism.

There’s even a scenario in which the field evolves in such a way to halt the expansion of the universe and cause it to collapse back on itself.

But this flexibility makes it hard to actually falsify the theory - it’s just too easy to come up with a behavior of the field that fits our observations.

That said, physicists are trying.

The most direct test would be to find the particles of this field, for example in one of our particle colliders.

But there hasn’t been a whiff of them yet, but that doesn’t tell us much.

If we could actually confirm a change in the density of dark energy then it would be strong support for quintessence, because that would refute the cosmological constant model, and quintessence is its main competitor.

The now fully operational James Webb Space Telescope will help by pushing our measurements of the expansion rate back billions of years from the current supernova measurements.

There are also suggestions that the quintessence field may cause a characteristic signature in the stuff of the cosmic microwave background clumps together.

But again, no firm conclusions yet.

Long story short - measuring the equation of state of the universe with increasing precision will teach us about its fabric, its origin, and its fate.

Either the cosmological constant problem is coincidental with a quintessentially consistent dark energy, or scalar quantum fields shift in a quintessence-saturated space time.