V. I. Mogilevskii

Let $\mathfrak H$ be a Hilbert space and let $A$ be a symmetric linear relation (in particular, a nondensely defined operator) in $\mathfrak H$. By using the concept of a boundary triplet for $A^*$ we characterize symmetric extensions $\widetilde A\supset A$ preserving the multivalued part of $A$. Such a characterization is given in terms of an abstract boundary parameter and the Weyl function of the boundary triplet. Application of these results to the Hamiltonian system $Jy'-B(t)y=\lambda\Delta(t) y$ enabled us to describe its matrix solutions generating the generalized Fourier transform with the nonempty set of respective spectral functions.

The main object of the paper is a Hamiltonian system $J y'-B(t)y=\lambda\Delta(t) y$ defined on an interval $[a,b) $ with the regular endpoint $a$. We define a pseudo\-spectral function of a singular system as a matrix-valued distribution function such that the generalized Fourier transform is a partial isometry with the minimally possible kernel. Moreover, we parameterize all spectral and pseudospectral functions of a given system by means of a Nevanlinna boundary parameter. The obtained results develop the results by Arov-Dym and Sakhnovich in this direction.

Let $H$ be a finite dimensional Hilbert space and let $[H]$ be the set of all li ear operators in $H$. We consider first-order symmetric system $J y'-B(t)y=\Lambda(t) f(t)$ with $[H]$-valued coefficients defined on an interval $[a,b) $ with the regular endpoint $a$. It is assumed that the corresponding minimal relation $T_{\rm min}$ has maximally possible deficiency index $n_+(T_{\rm min})=\dim H$. The main result is a parametrization of all characteristic matrices and pseudospectral (spectral) functions of a given system by means of a Nevanlinna type boundary parameter $\tau$. Similar parametrization for regular systems has earlier been obtained by Langer and Textorius. We also show that the coefficients of the parametrization form the matrix $W(\lambda)$ with the properties similar to those of the resolvent matrix in the extension theory of symmetric operators.

We study general (not necessarily Hamiltonian) first-ordersymmetric system $J y'-B(t)y=\Delta(t) f(t)$ on an interval $\mathcal I=[a,b)$ with the regular endpoint $a$ and singular endpoint $b$. It isassumed that the deficiency indices $n_\pm(T_{\min})$ of thecorresponding minimal relation $T_{\min}$ in $L_\Delta^2(\mathcal I)$ satisfy$n_-(T_{\min})\leq n_+(T_{\min})$. We describe all generalized resolvents$y=R(\lambda)f, \; f\in L_\Delta^2(\mathcal I),$ of $T_{\min}$ in terms of boundary problemswith $\lambda$-depending boundary conditions imposed on regular andsingular boundary values of a function $y$ at the endpoints $a$and $b$ respectively. We also parametrize all characteristicmatrices $\Omega(\lambda)$ of the system immediately in terms of boundaryconditions. Such a parametrization is given both by the blockrepresentation of $\Omega(\lambda)$ and by the formula similar to thewell-known Krein formula for resolvents. These results develop the Straus' results on generalized resolvents and characteristicmatrices of differential operators.

Let $A$ be a symmetric linear relation with arbitrary deficiency indices. By using the conceptof the boundary triplet we describe exit space self-adjointextensions $\widetilde A^\tau$ of $A$ in terms of a boundary parameter $\tau$. We characterize certain geometrical properties of $\widetilde A^\tau$ and describe all $\widetilde A^\tau$ with ${\rm mul}\, \widetilde A^\tau=\{0\}$. Applying these results to general (possibly non-Hamiltonian) symmetric systems $Jy'- B(t)y=\Delta(t)y, \; t \in [a,b\rangle,$ we describe all matrix spectral functions of theminimally possible dimension such that the Parseval equality holdsfor any function $f\in L_\Delta^2([a,b \rangle)$.

Let $l [y]$ be a formally selfadjoint differential expression of an even order on the interval $[0,b \rangle$, $b\leq \infty$, with operator coefficients, acting in a separable Hilbert space $H$. We introduce the concept of deficiency indices $n_{b\pm}$ of the expression $l$ at the point $b$ and show that in the case $\dim H=\infty$ any values of $n_{b\pm}$ are possible. Moreover the decomposing selfadjoint boundary conditions exist if and only if $n_{b+}=n_{b-}$. Our considerations of differential operators with arbitrary (possibly unequal) deficiency indices are based on the concept of a decomposing $D$-boundary triplet. Such an approach enables to describe extensions of the minimal operator directly in terms of operator boundary conditions at the ends of the interval $[0,b \rangle$. In particular we describe in a compact form selfadjoint decomposing boundary conditions.

Associated to a $D$-triplet is an $m$-function, which can be regarded as a gene alization of the classical characteristic (Titchmarsh-Weyl) function. Our definition enables to describe all $m$-functions (and, therefore, all spectral functions) directly in terms of boundary conditions at the right end $b$.

Let $H$ be a Hilbert space and let $A$ be a symmetric operator in $H$ with arbitrary (not necessarily equal) deficiency indices $n_\pm (A)$. We introduce a new concept of a $D$-boundary triplet for $A^*$, which may be considered as a natural generalization of the known concept of a boundary triplet (boundary value space) for an operator with equal deficiency indices. With a $D$-triplet for $A^*$ we associate two Weyl functions $M_+(\cdot)$ and $M_-(\cdot)$. It is proved that the functions $M_\pm(\cdot)$ posses a number of properties similar to those of the known Weyl functions ($Q$-functions) for the case $n_+(A)=n_-(A)$. We show that every $D$-triplet for $A^*$ gives rise to Krein type formulas for generalized resolvents of the operator $A$ with arbitrary deficiency indices. The resolvent formulas describe the set of all generalized resolvents by means of two pairs of operator functions which belongs to the Nevanlinna type class $\bar R(H_0,H_1)$. This class has been earlier introduced by the author.

Let H1 be a subspace in a Hilbert space H0 and let $\widetilde C(H_0,H_1)$ be the set of all closed linear relations from $H_0$ to $H_1$. We introduce a Nevanlinna type class $\widetilde R_+ (H_0,H_1)$ of holomorphic functions with values in $\widetilde C(H_0,H_1)$ and investigate its properties. In particular we prove the existence of a dilation for every function $\tau_+(\cdot)\in \widetilde R_+ (H_0,H_1)$. In what follows these results will be used for the derivation of the Krein type formula for generalized resolvents of a symmetric operator with arbitrary (not necessarily equal) deficiency indices.