A subset $S$ of a topological dynamical system $(X,f)$ containing
at least two points is called a scrambled set if for any
$x,y\in S$ with $x\neq y$ one has
$$\liminf_{n\to \infty} d(f^n(x), f^n(y)) = 0 \quad \hbox{and}
\quad \limsup_{n\to \infty} d(f^n(x), f^n(y)) > 0,$$ $d$ being the
metric on $X$. The system $(X,f)$ is called Li–Yorke chaotic
if it has an uncountable scrambled set.These notions were developed in the context of interval maps, in
which the existence of a two-point scrambled set implies Li–Yorke
chaos and many other chaotic properties. In the present paper we
address several questions about scrambled sets in the context of
topological dynamics. There the assumption of Li–Yorke chaos, and
also stronger ones like the existence of a residual scrambled set,
or the fact that $X$ itself is a scrambled set (in these cases the
system is called residually scrambled or completely scrambled
respectively), are not so highly significant. But they still provide
valuable information.First, the following question arises naturally: is it true in
general that a Li–Yorke chaotic system has a Cantor
scrambled set, at least when the phase space is compact?
This question is not answered completely but the answer is known
to be yes when the system is weakly mixing or Devaney chaotic or
has positive entropy, all properties implying Li–Yorke chaos; we
show that the same is true for symbolic systems and systems without
asymptotic pairs, which may not be Li–Yorke chaotic. More generally, there are severe
restrictions on Li–Yorke chaotic dynamical systems without a Cantor scrambled set, if they exist.A second set of questions concerns the size of scrambled sets
inside the space $X$ itself. For which dynamical systems
$(X,f)$ do there exist first category, or second category, or
residual scrambled sets, or a scrambled set which is equal to the
whole space $X$?While reviewing existing results, we give examples of systems on
arcwise connected continua in the plane having maximal scrambled
sets with any prescribed cardinalities, in particular systems
having at most finite or countable scrambled sets. We also give
examples of Li–Yorke chaotic systems with at most first category
scrambled sets. It is proved that minimal compact systems, graph
maps and a large class of symbolic systems containing subshifts of
finite type are never residually scrambled; assuming the Continuum
Hypothesis, weakly mixing systems are shown to have
second category scrambled sets. Various examples of residually
scrambled systems are constructed. It is shown that for any
minimal distal system there exists a non-disjoint completely
scrambled system. Finally, various other questions are solved. For
instance, a completely scrambled system may have a factor without
any scrambled set, and a triangular map may have a scrambled set
with non-empty interior.
