5.1Introduction

Motivation

Optical flow is the apparent motion in a sequence of images and can be described by a velocity field. Using variational techniques to estimate this velocity field requires the design of an appropriate energy functional. Typically such a functional is a sum of two parts. The first part, sometimes called data term, measures the accuracy with which the velocity field describes the observable image motion. By ensuring consistency of the flow field, the second part, also called regularization term, gives information the data term cannot provide and thereby addresses the inherent ill-posedness of the optical flow problem. Naturally, related research has to a large degree been concerned with tuning the second term to optimally capture the characteristics of velocities of real-world image sequences.

Generally speaking regularization terms for optical flow fall into two categories: those which only penalize spatial derivatives of the velocity field, and those where derivatives in both space and time are penalized. The first category is by far the more popular one. Its developments have closely followed those of variational image restoration, because the regularity of images dictates to a significant degree the spatial regularity of velocity fields describing their motion. In other words, image discontinuities tend to coincide with motion discontinuities. These, it was found, can be appropriately dealt with using subquadratic or anisotropic regularization.

Research on the second category, that is that of spatiotemporal regularization, is relatively scarce. Again this is in analogy with the field of image processing, where the joint restoration of image sequences is quite under-represented as compared to the restoration of single frames. The inclusion of time derivatives has the obvious disadvantage of turning a series of decoupled 2D problems into one 3D problem. In the early days of computer vision limited computer memory and processing power was prohibitive to such approaches. Clearly this is no longer the case. On the contrary, recent publications have articulated and addressed the need for time-coupled models even for data with three instead of two space dimensions. See for example [2, 15].

It is well known that indiscriminate smoothing in space can lead to blurring at the boundaries of objects. This statement is equally true in the space-time domain, if we identify objects with their space-time signature (Figure 1). In particular, blind penalization of the partial time derivative of a velocity field can lead to a loss of accuracy, especially at the boundaries of moving objects. Therefore, we suggest to take the time derivative only along the movement of the object.

This derivative has a very natural physical interpretation. Let υ(t, x) be a timedependent velocity field on ℝd, and let t ↦ γ(t; x0) be the trajectory of a certain object initially located at x0 ∈ ℝd that moves according to υ. The translation into mathematical terms of this connection between γ and υ is the following initial value problem

Taking time derivatives on both sides of the differential equation above gives a formula in terms of υ for the acceleration of the object moving along γ

This expression, which is sometimes called convective acceleration, is to be contrasted with the partial time derivative of υ evaluated at (t, γ(t; x0))

which has a notably different physical meaning. In this article we argue that in some situations the former can be more appropriate for the regularization of optical flow.

This work is devoted to investigating the suitability of the functional

as a smoothness term for optical flow. We believe a sensible way to do so is to include this term into a simple optical flow model and to compare the results with those of the same model without the new term. The reference model we choose for this task is a contrast invariant and time-regularized version of the Horn–Schunck functional [7]. Incorporating (1.2) into our model comes, however, at the cost of leading to a nonconvex functional and to nonlinear optimality conditions. Numerically, we approximate the problem by a sequence of convex quadratic ones.

We stress that it is not the aim of the proposed model to compete in the highest ranks of motion estimation benchmarks. Instead we want to point out a certain aspect of optical flow regularization, which we feel has not received its due amount of attention. Even though it leads to a more challenging variational problem, we demonstrate that a reasonable approximation is numerically tractable and pays off in terms of accuracy.

Related work

Horn and Schunck [7] are generally credited with having laid the groundwork for variational optical flow models. Penalizing the squared L2-norms of the spatial derivatives, their model regularizes the optical flow isotropically as well as homogeneously over the image domain. In [10], Nagel proposed suppression of smoothing across image discontinuities by only penalizing the derivatives of the vector field along the level lines of the image. This idea was subsequently extended to the space-time domain in [11].

A different route to spatiotemporal regularization was taken by Weickert and Schnörr. In [19] they used an isotropic but essentially subquadratic regularizer.While still convex this approach leads to nonlinear optimality conditions. In the survey [18], the same authors classify convex spatial as well as spatiotemporal regularizers for optical flow. According to their nomenclature our approach would classify as anisotropic and flow-driven. It is, however, fundamentally different from the regularizers considered there, not only because of nonconvexity. In [18] the anisotropy of flow-driven regularizers is determined by the Jacobian of the velocity field, whereas in our approach it is determined by the velocity itself.

Chaudhury and Mehrotra pursue an interesting trajectory-based regularization strategy [3]. Motivated by the principle of least action and the inertia of motion they postulate that both length and curvature of trajectories of moving objects should be minimal. There is a close connection between a trajectory’s curvature and the convective acceleration of an object moving along that trajectory. We discuss this connection in the next section. More recently, Salgado and Sánchez [14] as well as Volz and coauthors [17] have proposed time-discrete models using explicit trajectorial regularization. The present article tries to capture the essence of trajectorial regularization for optical flow in an entirely continuous setting.

Finally, we note that for the problem of image sequence reconstruction it is not uncommon to regularize the sequence along the optical flow, which can be either precomputed or estimated simultaneously. This derivative along the optical flow is just the convective derivative of the image sequence. See [4, 12] for example.

Outline

In Section 5.2 we introduce our model. In Section 5.2.1 we discuss the convective acceleration of vector fields. Section 5.2.2 contains properties of the functional as a regularization term. The last part of Section 5.2 treats our choice of data term. Section 5.3 is dedicated to discussing our numerical minimization approach. Finally, we present experimental results in Section 5.4.

5.2Model

5.2.1Convective acceleration

Notation

Let Ω ⊂ ℝ2 be a bounded Lipschitz domain, T > 0, and set E = (0, T) × Ω. We denote points in Ω by x = (x1, x2)⊤. Let ϕ: E → Ω be a flow on Ω: That is, for fixed t ∈ (0, T) the map ϕ(t, ⋅) is a diffeomorphism of Ω,whilefor fixed x0 ∈ Ω the trajectory described by ϕ(⋅, x0) is smooth. The vector field on E that gathers the velocities of all trajectories associated to ϕ is defined by

where ϕt is the partial derivative of ϕ with respect to its first argument and ϕ−1(t, x) denotes the inverse of ϕ(t, ⋅) evaluated at x. Notice that u(t, x) describes the velocity of the trajectory passing through x at the time t, so u(s, x) in general corresponds to a different trajectory, if s ≠ t.

Let f : E →ℝN be a possibly vector-valued quantity on E.Wedefine the convective derivative of f along ϕ as

Here ∇f = (fx1 , fx2) ∈ ℝN×2 is the spatial gradient of f .Notice that the convective derivative only depends on u(t, x), thereby justifying the notation Duf . Using the notation ∇̄ f = (ft , ∇f) and ū = (1, u⊤)⊤ we can write Duf = ∇̄ f ū. If we set f = u, the resulting vector field

is called the convective acceleration of the flow ϕ.

Flows with vanishing convective acceleration

When proposing a new regularization term, it is always helpful to know when it is minimal. Therefore, we briefly discuss what conditions must be met so that a vector field u satisfies

and give a few examples afterwards. Intuitively Duu = 0 means that u satisfies a “flow constancy condition”. This is to be understood as an analogy to the widely known brightness constancy condition, which is frequently used to derive the optical flow equation. Note that with the notation introduced in the previous paragraph, for an image sequence f , the optical flow equation reads Duf = 0.

From the definition of the convective derivative (2.2) and (2.1) it follows that

Duu(t, ϕ(t, x0)) = ϕtt(t, x0) ,

for all x0 ∈ Ω and t ∈ (0, T). Now fix an x0 ∈ Ω and, for simplicity, denote the trajectory ϕ(⋅, x0) by γ(⋅), so that we have Duu = γ″. Assuming that γ′ does not vanish on (0, T), we can write

Since the unit tangent vector γ′/|γ′| is always perpendicular to its derivative, the convective acceleration γ″ is a sum of two mutually orthogonal vectors. Such a sum can only vanish, if both vectors vanish. Because of the assumption that γ′ ≠ 0, we conclude that

This proves the quite intuitive fact that, if a flow has vanishing convective acceleration, then all its trajectories must be straight lines and have constant speed. Below we give a few examples of such vector fields.

Example 2.1. Let g : ℝ → ℝ be a differentiable function. The two vector fields υ1, υ2 : E → ℝ2 defined by

do not depend on time and satisfy ∇υiυi = 0. Thus, by formula (2.3) they have vanishing convective acceleration. Their integral curves are parallel to the x2- and x1-axes, respectively.

Example 2.2. Let 0 ∉ Ω. Denote by er the unit vector in radial direction and let g : ℝ → ℝ be a differentiable function with period 2π. The vector field υ: E → ℝ2 which in polar coordinates is given by

υ(t, r, φ) = g(φ)er

has zero convective acceleration. In contrast to the previous example the integral lines of υ are not all mutually parallel.

Example 2.3. The vector fields in the previous examples were all constant in time. Time-dependent examples can be constructed from time-dependent solutions u of the inviscid Burgers’ equation in one space dimension ([6], Section 3.4). That is, let u : E → ℝ solve (2.4) with E = (0, T) × ℝ. Then

satisfy Dυi υi = 0 while having partial time derivatives that do not vanish identically. As in the first example the integral curves are mutually parallel straight lines. The main difference, however, is that two trajectories passing through the same point in space but at different times, might do so at different speeds.

Relation to curvature

Given a curve γ : (0, T) → Ω, we can define its curvature as the normal part of the arc length derivative of the unit tangent vector

where (a, b)⊤⊥ = (−b, a)⊤. If γ is arc length parametrized, that is |γ′| ≡ 1, then the expression simplifies to κγ = |γ″|, because in that case γ′⊥ = γ″/|γ″|.

For a given x0 ∈ Ω we can apply the formula above to the trajectory t ↦ ϕ(t, x0) of a flow ϕ. Recall that in this case γ′ becomes u and γ″ becomes Duu. Thus, we get

In particular, whenever |u(t, ϕ(t, x0))| ≡ 1 on some subinterval of (0, T), then the norm of the convective acceleration |Duu| coincides with the absolute value of the curvature of ϕ(⋅, x0) on this subinterval

5.2.2Convective regularization

In this section we study some properties of the functional

as a regularization term for optical flow. By ‖⋅ ‖L2 we mean the norm of L2(E, ℝ2). Below we continue to use this shorthand whenever convenient.

Interpretation of the convective term

Functional (2.5) has two interpretations, one as a smoothing term and another one as a projection term. Recalling the bar-notation we introduced in Section 5.2.1, the integrand of (2.5) can be written as a quadratic form in the partial derivatives of u

Observe that the diffusion tensor

is a projection matrix onto the span of ū composed with a scaling by |ū|2. Therefore, minimization of (2.5) leads to smoothing of the vector field u only in direction ū, where precedence is given to regions where the magnitude of u is relatively large. Clearly, since ū ≠ (1, 0, 0)⊤ in general, the proposed convective regularization term is not a purely temporal regularizer, but it also enforces spatial smoothness of the vector field in a way that is consistent with the motion. Figure 2 illustrates this behavior.

Conversely, the roles of ū and ∇̄ui can be exchanged so that (2.6) rewrites as a sum of two quadratic forms in ū

The matrix (∇̄ui)⊤∇̄ui projects onto the span of ∇̄ ui and scales by the squared magnitude of ∇̄ui. From this point of view minimizing (2.5) amounts to forcing ū to be orthogonal to the directions of greatest change of both u1 and u2, where precedence is again given whenever this change is relatively drastic. Note that the outer product of a vector with itself is always a positive semidefinite matrix, so that there can be no cancelations in (2.6), (2.7).

The concurrence of the two different interpretations above is made explicit by looking at the Euler–Lagrange equation of (2.5), given by

Here, the space-time divergence operator ∇̄ ⋅ acts along rows.

Variational properties

In the following we adopt the notation A ≲ B whenever A ≤ B holds up to multiplication by a positive constant. According to the basic estimate

a seemingly natural space over which to minimize, for a given image sequence f, the optical flow functional

would be X = H1(E, ℝ2) ∩ L∞(E, ℝ2). Now, in order to show existence of minimizers using the direct method [5], we need to ensure existence of a minimizing sequence which converges. This is typically done by establishing a coercivity condition of the type

for some b ≥ 0 and p ≥ 1. In the examples above we have seen, however, that there are vector fields u with vanishing convective acceleration but nonzero partial derivatives. This implies that not even the inequality

holds true. Therefore, in order to guarantee existence of minimizers we consider thefunctional

where α, β > 0 and λ : E → ℝ+ is a weighting function which is specified in Section 5.2.3. In addition we have to make the assumption introduced in [16] that the partial derivatives of f are linearly independent in L2(E), that is

Proposition 2.4. Let f ∈ W1,∞(E) satisfy (2.10) and assume that λ ∈ L∞(E) is such that ess inf λ > 0. Then the minimization of ε over H1(E, ℝ2) has at least one solution.

Proof. We show that ε is proper, coercive and lower semicontinuous with respect to the weak topology of H1(E, ℝ2). That ε is proper follows from nonnegativity and the assumptions on f and λ.

Next, we prove the coercivity estimate

for b ≥ 0.We obviously have ε(u) ≥ β‖ ∇̄ u‖2 . So it remains to show that b. Denoting the average of u by uE = 1/|E| ∫E u we have

which proves coercivity. There are three main ingredients in the above estimate. The first one is a quadratic variant of the triangle inequality

‖υ + w‖2 ≤ 2‖υ‖2 + 2‖w‖2 .

The second one

uses the assumption of linear independence of fx1 and fx2 and is proved in [16], p. 29. The third ingredient is the Poincaré inequality ‖u − u E‖L2 ≲ ‖∇̄u‖L2 ([9], Theorem 8.11). Also note that the fourth inequality in (2.11) requires the assumption f ∈ W1,∞(E), while the last one uses ess inf λ > 0.

Let {un} ⊂ H1(E, ℝ2) converge to û in the weak topology of H1(E, ℝ2). In particular, ∇̄un converges weakly in L2(E, ℝ6) to the corresponding gradient ∇̄û. By the compact embedding of H1(E, ℝ2) into L2(E, ℝ2), up to choosing a subsequence we may assume that un also converges strongly in L2 to û. As is made apparent in (2.6), the expression |Duu|2 is convex in ∇̄u for fixed u, and ε is always nonnegative. We can therefore apply a standard result ([5], Theorem 3.23) to obtain that ε is lower semicontinuous.

There is no reason to expect minimizers of ε to be unique. The following example demonstrates that ε is not convex in general.

Example 2.5. Let υ1, υ2 be the two vector fields from Example 2.1 with g being the identity, that is

They satisfy Dυ1υ1 = Dυ2υ2 ≡ 0, while

where w = (υ1 + υ2)/2.We conclude

so that the functional cannot be convex. This clearly implies that there are α, β, f such that ε is not convex.

5.2.3Data term and contrast invariance

Contrast invariance is a useful property of image processing operators. Let h be a change of contrast, that is, a differentiable real function with h′ > 0. An operator A is called contrast invariant, if it commutes with all such contrast changes:

A ∘ h(f) = h ∘ A(f)

for all images f ([1], Section 2.3).

A similar property can be postulated for operators estimating image motion, since an order-preserving rearrangement of the grady values of an image sequence should certainly not change velocities. Therefore, we call an operator A that maps image sequences f to velocity fields contrast invariant, if it satisfies

A(f) = A ∘ h(f) .

However, a typical optical flow model of the form

does not have this property. A simple counterexample is multiplication by a positive number h(f) = cf . For a model like (2.12) this change of contrast effectively amounts to dividing the regularization parameter α by cp and therefore definitely influences the result. There is, however, also a local effect of this contrast dependence, which is easy to confirm experimentally. Consider a scene with a dark background and two moving objects which are similar in shape and size and which move at similar velocities, but which have significantly different average brightnesses. This is precisely the case in the example of Figure 3. Then, the velocities of the darker object, that is the one with lower values of f , will be regularized more strongly than those of the brighter one, if a data term like the above is used.

However, contrast dependence is not an inherent property of optical flow. Observe that, if u solves Duf = 0, then it also solves Du(h∘ f) = 0, because Du(h∘ f) = h′Duf and h′ > 0 by assumption. Therefore, it should rather be regarded as a side-effect of the variational regularization approach. Contrast invariance can be restored by weighting the data term with λ = 1/ω, where ω is a positively 1-homogeneous function that depends only on the first-order derivatives of f . It then follows that

Such weighted data terms have already been used, although itseems with a different reasoning. In [8, 20], for example, theweight was chosen to be Inthis paper we experiment with the weight With this choice the data term essentially measures the orthogonal projection of ū onto the span of ∇̄ f , since

The effectiveness of this choice is illustrated in Figure 3, in which the relative velocities of the vehicles are recovered correctly even though they have widely different contrast levels.

Finally, observe that weights which do not depend on first derivatives of f (only), like ω = |f| + ϵ for instance, do not yield full contrast invariance. While they work fine for the special case of rescaling h(f) = cf , they fail in general since the factor h′ in (2.13) does not cancel.

5.3Numerical solution

We now seek to formulate a numerical approach for the minimization of the functional ε. As shown above, the convective regularization term is nonconvex and leads to nonlinear optimality conditions. Therefore, we propose an iterative scheme to arrive at an adequate flow field.

Iterative scheme

Consider the functional

which satisfies ε(u) = ?(u, u). For fixed w ∈ L∞(E, ℝ2), the mapping u ↦ ?(u, w) is a convex and quadratic functional, and ?(u, w) < +∞for any u ∈ H 1(E). Our iterative method then reads as follows.

Each of these steps corresponds to finding an optical flow field using anisotropic regularization with the diffusion tensor αw̄ w̄ ⊤ + βId, where Id is the 3×3 identity matrix. Therefore, this particular scheme is based on the diffusion interpretation of the convective regularization term, as reflected in (2.6). Indeed, upon fixing w the first term in the complete Euler–Lagrange equation (2.8) does not appear. Even though convergence of the iterative scheme is not guaranteed, in all of our experiments the solution u k stabilized after only a few iterations. Typically by k = 3 or 4 the difference between successive iterates became negligible.

By the properties of ? listed above, each of these minimization problems can be performed by solving the corresponding linear Euler–Lagrange equation

coupled with natural boundary conditions

where n is the outward normal to E. Our approach of discretizing (3.1) is specified next.

Discretization

The model has been discretized using a multilinear finite element formulation on a regular rectangular grid, so that each vertex in the grid corresponds to one pixel of one video frame. The grid used is such that the spacing between nodes in space is one, while the spacing in time (from frame to frame) may be smaller. We emphasize that the relation of the two grid step sizes implicitly controls the relative amount of regularization done in time and space, since the derivatives of the functional scale accordingly. Usually, using the same spacing will lead to results that are highly overregularized in time. In the presented examples, the time step was divided by a factor of 8 with respect to that of space.

Both the input f and the unknown vector field u are represented by their nodal values, leading to continuous piecewise multilinear functions. Since the Euler–Lagrange equation (3.1) is actually a system for the two components u1, u2, the resulting linear system is represented by a block matrix, with the different blocks describing the interactions between components. The entries of these blocks are assembled from the integrals over different elements arising in the weak formulation, as per finite element practice [13]. Such integrals are computed exactly through an adequate Gauss quadrature. Note that boundary conditions do not need to be imposed explicitly.

A consequence of representing f by a piecewise multilinear function is that the corresponding partial derivatives fxi , ft appearing in the right hand side λ2ft∇f⊤ of (3.1) do not belong to the same finite element space as f . However, our finite element matrices act on the nodal values of functions in this space, so an interpolation step is needed to obtain functions with the appropriate smoothness. To this end, we have performed an L2 projection of these partial derivatives into the space of piecewise multilinear functions, whose nodal values are then multiplied together and used as the right hand side for our linear system. Computationally each projection requires the solution of an additional linear system, which needs to be done only once.

5.4Experiments

In Figures 4, 5 and 6 we present some examples of flows computed through the numerical scheme of the previous section, approximating minimizers of

which are compared with corresponding results with basic isotropic regularization, that is, minimizers of

In all these examples results were computed over an interval of 30 frames.

In the sequences used, the time derivative of the vector field at a given position is likely not appropriate, since isolated objects travel across the images. In the results with isotropic regularization, some of the disadvantages of basic quadratic regularizers are apparent. With lower regularization parameters (Figures 4b and 5b) inner edges of rigidly moving objects are visible. With higher regularization parameter values, the support of the vector field is enlarged and interactions between distinct objects moving ensue, creating cancelations when the objects move in different directions (Figure 4c) or artificial reinforcement when they move in similar directions (Figure 5c). However, with convective regularization both of these disadvantages can be avoided (Figures 4d and 5d).

Additionally, one needs to choose the regularization constant ϵ for the weight in the data term. Clearly, choosing a too large ϵ will diminish the effect of the weighting scheme, while choosing it too small may overemphasize very low contrast regions and potentially amplify noise. Since the 8-bit input images were normalized so that their values are of the form k/255 with k = 0 . . . 255, we chose ϵ = 0.01. That is, ϵ is roughly of the size of the smallest nonzero derivative that may appear in the data.

In our experiments the parameter β1 may be chosen quite small to preserve the boundaries of objects, while using a relatively large α1 for the convective regularization term. By using the convective derivative, the information from other frames is used in a consistent way to correctly fill the flow field inside objects even in the absence of texture. The initial isotropic parameter β0 for starting the iterative scheme was chosen to be β0 = α1 in all cases.

5.5Conclusion

In this article we presented an entirely continuous optical flow model that is based on the assumption that velocities should vary smoothly not at fixed points in space but along motion trajectories. The resulting regularization term has a variety of different interpretations. First, this term penalizes the proper acceleration of objects moving according to the velocity field. Second, it has a direct relation to the curvature of motion trajectories. Finally, whenminimized it acts on the vector field not only as a projection term but also as an edge-preserving anisotropic regularizer.

We tested the convective regularization term by incorporating it into a reference optical flow model and comparing the results with and without the new term. In order to make the changes as apparent as possible and to avoid potential side effects we triedto keep the complexity of the reference model at a minimum. Therefore, the resulting functional (2.9) should mainly be viewed as a prototype of an optical flow model with convective regularization which can be improved upon by future research.

Acknowledgments

This work was supported by the Austrian Science Fund (FWF) within the national research network Geometry + Simulation (project S11704, Variational Methods for Imaging on Manifolds). Additionally, J. A. Iglesias acknowledges support from the FWF Doctoral Program Dissipation and Dispersion in Nonlinear PDEs (W1245). We wish to thank Enrique C. Vargas for inspiring exchanges. Our finite element implementation was carried out in the Quocmesh library,14AG Rumpf, Institute for Numerical Simulation, Universität Bonn. For our experiments, we have used footage from the datasets of Lund University15(sequence “Conflicts-20100628-1344, camera 1”) and Universität Karlsruhe16(sequences “taxi” and “dt_passat_part2”).

Fig. 5: Comparison of isotropic and convective regularization.

Fig. 6: Cutout from Figure 5. The use of convective regularization allows us to obtain a nearly constant optical flow field inside the objects without excessively enlarging the support.

Bibliography

[1]L. Alvarez, F. Guichard, P.-L. Lions, and J.-M. Morel. Axioms and fundamental equations of image processing. Arch. Ration. Mech. Anal., 123(3):199–257, 1993.

[2]F. Amat, E.W. Myers, and P. J. Keller. Fast and robust optical flow for time-lapse microscopy using super-voxels. Bioinformatics, 29(3):373–380, 2013.

[3]K. Chaudhury and R. Mehrotra. A trajectory-based computational model for optical flow estimation. IEEE Trans. on Robotic. Autom., 11(5):733–741, Oct 1995.

[4]J. P. Cocquerez, L. Chanas, and J. Blanc-Talon. Simultaneous inpainting and motion estimation of highly degraded video-sequences. In J. Bigun and T. Gustavsson, editors, Image Analysis, volume 2749 of Lecture Notes in Computer Science, pp. 685–692. Springer Berlin Heidelberg, 2003.

[5]B. Dacorogna. Direct methods in the calculus of variations, volume 78 of Applied Mathematical Sciences. Springer, New York, 2nd edn., 2008.

[6]L. C. Evans. Partial Differential Equations, volume 19 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2nd edn., 2010.

[7]B. K. P. Horn and B. G. Schunck. Determining optical flow. Artificial Intelligence, 17:185–203, 1981.

[8]S.-H. Lai and B. C. Vemuri. Reliable and efficient computation of optical flow. Int. J. Comput. Vision, 29(2):87–105, 1998.

[9]E. H. Lieb and M. Loss. Analysis, volume 14 of Graduate Studies in Mathematics. American Mathematical Society, 2nd edn., 2001.

[10]H.-H. Nagel. Constraints for the estimation of displacement vector fields from image sequences. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence,IJCAI 83, pp. 945–951, 1983.

[11]H.-H. Nagel. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In Proceedings of the First European Conference on Computer Vision, ECCV 90, pp. 139–148, New York, NY, USA, 1990. Springer-Verlag New York, Inc.

[12]T. Preusser, M. Droske, C. S. Garbe, A. Telea, and M. Rumpf. A phase field method for joint denoising, edge detection, and motion estimation in image sequence processing. SIAM J. Appl. Math., 68(3):599–618, 2008.

[13]A. Quarteroni and A. Valli. Numerical Approximation of Partial Differential Equations, volume 23 of Springer Series in Computational Mathematics. Springer Verlag, Berlin, 2008.

[14]A. Salgado and J. Sánchez. Temporal constraints in large optical flow estimation. In R. Moreno-Díaz, F. Pichler, and A. Quesada-Arencibia, editors, Computer Aided Systems Theory – EURO-CAST 2007, volume 4739 of Lecture Notes in Computer Science, pp. 709–716. Springer Berlin Heidelberg, 2007.

[15]B. Schmid, G. Shah, N. Scherf, M. Weber, K. Thierbach, C. Campos Pérez, I. Roeder, P. Aanstad, and J. Huisken. High-speed panoramic light-sheet microscopy reveals global endodermal cell dynamics. Nat. Commun., 4:2207, 2013.

[16]Ch. Schnörr. Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class. Int. J. Comput. Vision, 6:25–38, 1991.

[17]S. Volz, A. Bruhn, L. Valgaerts, and H. Zimmer. Modeling temporal coherence for optical flow. In IEEE International Conference on Computer Vision (ICCV), pp. 1116–1123, Nov 2011.

[18]J. Weickert and C. Schnörr. A theoretical framework for convex regularizers in PDE-based computation of image motion. Int. J. Comput. Vision, 45(3):245–264, 2001.

[19]J. Weickert and Ch. Schnörr. Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vision, 14:245–255, 2001.

[20]H. Zimmer, A. Bruhn, and J. Weickert. Optic flow in harmony. Int. J. Comput. Vision, 93(3):368–388, 2011.