/articles/adei2016

\documentclass[a4paper]{jpconf}
\bibliographystyle{iopart-num}
\usepackage{citesort}
\usepackage{graphicx}
\usepackage{enumitem}



\begin{document}
\title{Application multivariate analysis and visualization tools in ADEI for studies of the high-energy phenomena in atmosphere}

\author{A Chilingarian$^1$, S Chilingaryan$^2$, H Gemmeke$^2$ A Kopmann$^2$, A Reymers$^1$}
\address{$^1$ Yerevan Physics Institute, Armenia}
\address{$^2$ Karlsruhe Institute of Technology, Germany}

\ead{chili@aragats.am}

\begin{abstract}
To make transformational scientific progress in Space science and geophysics, the Sun, heliosphere, magnetosphere and different layers of the atmosphere must be studied as a coupled system. 
Presented paper describes how information on complicated physical processes on Sun, in heliosphere, magnetosphere and atmosphere can be made immediate assessable for researchers via advanced multivariate visualization system with simple statistical analysis package. Research aimed on studies of the high-energy phenomena in atmosphere and the atmospheric discharge is of special importance. The problem of how lightning is initiated inside thunderclouds is not only one of the biggest unsolved problems in lightning physics; it is also probably one of the biggest mysteries in the atmospheric sciences. The relationship between thundercloud electrification, lightning activity, wideband radio emission and particle fluxes have not been yet unambiguously established. One of most intriguing opportunities opening by observation of the high-energy processes in the atmosphere is their relation to lightning initiation. Investigations of the accelerated structures in the geospace plasmas can as well shed light on particle acceleration to much higher energy in the similar structures of space plasmas in the most distant objects in the universe. 
\end{abstract}

\section{Introduction}
In recent years, the interest in using cosmic rays for obtaining information on atmospheric and extra-atmospheric processes is rapidly growing. Cosmic rays are modulated by the solar activity and can be used as messengers carrying information on upcoming space storms. Precise and continuous monitoring of the secondary cosmic rays with surface networks of particle detectors can disentangle the modulation effects posed by solar activity on the galactic cosmic rays. The methodology of the recovery of strength and danger of agents of solar activity (Interplanetary coronal mass ejections and solar energetic proton events) was developed at the Cosmic Ray Division (CRD) and tested on the violent events of the 23-rd solar activity cycle (1997-2008, Chilingarian, 2009, Bostanjyan & Chilingarian, 2009, Chilingarian & Bostanjyan, 2010, Mailyan & Chilingarian 2010, Hovhannisyan & Chilingarian, 2011, Chilingarian & Karapetyan, 2011). Recently it was discovered that fluxes of cosmic rays detected on the earth’s surface also carry information on the parameters of the atmosphere, primarily on very difficult to measure atmospheric electricity (Chilingarian et al., 2010, 2011, 2012a). Fluxes of gamma rays and muons carry information on high-energy processes in the atmosphere and on net potential of atmospheric electric fields associated with emerging positive and negative charged layers in thunderclouds. Fluxes of the “thunderstorm” neutrons, first reliably detected on Aragats (Chilingarian et al., 2012b and 2012c) are connected with the photonuclear reactions of the gamma rays with atmospheric nuclei and can pose radioactive hazard to crew and passengers of the nearby aircrafts (Drozdov et al., 2012). The muon flux provides the main contribution to the natural background ionizing radiation at the Earth's surface and, in the absence of disturbances in the interplanetary magnetic field; the magnetosphere and atmosphere represent a stable radiation source. Cosmic ray muons come to the observation point from all directions of the upper celestial hemisphere and are sensitive to any changes in the flux of primary cosmic rays and the meteorological conditions of the atmosphere. A thorough study of all aspects of the impact of atmospheric and extra-atmospheric processes on cosmic rays (see Chilingarian & Mkrtchyan, 2012d), and the solution of the inverse problem – definition of the characteristics of these processes on registered variations in cosmic rays - require as complete measurement of various component of cosmic rays as possible.

One of the recognized leaders in contemporary investigations of geophysical phenomena using cosmic ray detection is the Cosmic Ray Division of the A.I. Alikhanyan National Scientific Laboratory of Armenia and its Aragats Solar Enviromental Center (ASEC).  At CRD’s Aragats and Nor Amberd mountain stations the networks of detectors registering electrons, muons, gamma rays and neutrons operate round the clock, providing important information on various geophysical processes. Methods for visualization and analysis of multi-dimensional experimental data developed in the laboratory are successfully used to research solar terrestrial connections and high-energy phenomena in the terrestrial atmosphere. Recently precise devices to measure magnetic and electrical fields, meteorological conditions, lightning occurrence location and determination of its type were placed at the Aragats and Nor Amberd research stations. Data from this instrumentation and the associated research gave us important information about the fluxes of electrons and gamma rays from thunderclouds.  These particles and the penetrating radiation from thunderclouds, on the local changes of electric and magnetic fields and its relationship to electrical fields in thunderclouds and other key metrological parameters. Multivariate analysis of variations of fields, radiation and fluxes can provide new information on the development of thunderstorm anomalies in the atmosphere, including those of catastrophic nature.
Such analysis presents a challenge due to the large quantity of measured parameters.   Huge amount of time series should be processed and identified near on-line for forecasting and alerts, as well as for report and paper preparations. Usually researchers have no time to access archives if data stream is pressing and new interesting events appear each new day. Therefore, to support researcher in statistical analysis and finding “new physics” the multivariate visualization platform should be supplemented with tools of elementary statistical analysis (histograms, moments, correlations, comparisons); figure preparation; archiving, i.e. a data exploration system.
    
Therefore, we supply the online stream of “big” data from ASEC with an intellectual exploration system developed in a collaboration between Karlsruhe Institute of Technology (KIT) and CRD. The Advanced Data Extraction Infrastructure (ADEI) helps researchers in exploring and understanding solar-terrestrial connections, solar modulation effects as well as in understanding high-energy phenomena in the atmosphere. A user-friendly interface interactively visualizes the multiple time-series and selects relevant parameters for different research objectives. Time series from different domains are joining to make a multivariate correlation analysis. The developed software links a multitude of space and geophysical observations into an integrated system that provides analysis tools and services to fully utilize the scientific potential of current and future space weather/geophysical observations. In this way we try to fully utilize the new concept of “big” data when an enormous amount of relevant observations culminates in the “new” physics unprecedented fast and precise. In this paper we will focus on the new options of the ADEI that allows not only on-line displaying the operation of measuring channels but also on-line analysis of the physical phenomena under investigation. 

\section{ADEI}
Advanced Data Extraction Infrastructure (ADEI) has been developed to provide data
exploration capabilities to a broad range of physical experiments dealing with time series. 

All these systems have very different characteristics: amount of data channels,
their types, sampling rates, etc. The data is stored in many different
ways utilizing various data formats and underlying database engines. 
On the other side users need information in different data formats which are
supported by analysis tools they use for post processing. 
Besides, operators need a tool providing possibility to examine all 
collected data checking the integrity and validity of measurements.
It is also needed to search and export data possessing specified characteristics.

To provide such broad coverage ADEI utilizes highly modular architecture. The 
system consists of backend and frontend parts communicating over HTTP protocol
using Asynchronous JavaScript and XML (AJAX~\cite{ajax}) approach. The ADEI backend defines
few abstract interfaces which are used to implement various capabilities using
simple plugins. The data sources are interfaced with dedicated drivers
implemented data access abstraction layer. 
The higher levels of system are relaying on this abstract interface 
to get data in a uniform way from arbitrary storage. 

The ADEI web frontend is inspired by \emph{GoogleMaps} interface. 
Single or multiple time series are plotted using the data from currently selected 
time interval. Then, the plot could be dragged and zoomed over time and value axes. 
The region of plot may be selected for detailed statistical analysis or exported in 
one of the supported formats. 

\subsection{Architecture}
ADEI is designed to deal with the data sampled at high rates and stored for long periods of time. The time span of measurements at ASEC goes back for 20 years and the fastest detectors are sampling the data at rates exceeding 10 Hz. Processing such amounts of data requires enormous computational power. However, the interactive tools should operate in near real-time and extract important information from this enormous amount of data. To achieve this goal ADEI continuously monitors incoming data, performs preprocessing, and caches important information in a high performance database. 

The simplified diagram of ADEI architecture is presented on Figure~\ref{arch}. The main logic of ADEI system
is contained in a backend which is implemented purely in PHP programming language. The backend incorporates
a data access layer, a caching daemon, an ADEI library. Communication with the web frontend and other client 
applications is maintained using web services. HTTP protocol is used for data exchange, XML for data encoding. 

\begin{figure}
\includegraphics[scale=0.50]{img/adei-arch.eps}
\caption{\label{arch} Architecture of Advanced Data Extraction Infrastructure. Data Source Access Layer unifies access
to the time series stored in different formats. After data filtering and quality checks the data is aggregated and stored in 
intermediate caching database. Access to the data is provided by ADEI library and web services are used to communicate with
client applications. }
\end{figure}

The data access layer hides details of underlying data sources providing other components of the system with a uniform way of 
data access. The data is organized hierarchically. The top level of hierarchy is the data source and ADEI may underline several 
data sources. The time series provided by the data source is divided in time-synchronized groups, so called \emph{LogGroups}. 
The current version includes modules to access data stored in relational databases accessible through PDO or ODBC interfaces,
several NOSQL databases, RRD (Round Robin Database Tool~\cite{rrd}) data format used by many system monitoring applications. 
Most of popular databases including MySQL/MariaDB, PostgreSQL, Oracle, Microsoft SQL server, and CoucheDB are supported.
 
The caching daemon is continuously running on a backend server and polls all data sources for a new data. When the data 
is acquired it piped through series of filters which check the data quality, apply correction coefficients and drop invalid
data. Then, the data is aggregated over intervals of few different periods. 
For each period, called \emph{cache level}, statistical information is gathered 
and stored in MySQL database (\emph{caching database}) as additional time series.
Minimum, maximum, and arvergage values, the total number of recorded records, and the 
amount of invalid or missing records for each interval of aggregation are stored. 
Then, this caches are used by ADEI to speed-up searches or provide data averaged over
the specified intervals. For instance, the plot module first selects the maximal cache 
level providing enough points to generate the plot of the specified size. The time resolutions 
of caching database are selected in the way that between 1000 and 10000 data samples can be extracted for any specified interval.
Such an amount of points fulfills most of plotting demands and in the same time the plots could be generated relatively fast.
After selection of cache level is made, the data is extracted from correspondent caching tables and one of supported algorithms is used to 
convert aggregated values into the graphic points. The data plots are generated with JpGraph on the backend and delivered to frontend as PNG images~\cite{jpgraph}. 

ADEI provides the stored data in multiple formats. We currently support CSV (Comma Separated Values), 
Microsoft Excel, NetCDF, ROOT (an analysis framework for high energy physics~\cite{root}),
and TDMS (Technical Data Management Streaming~\cite{tdms}). 
Additional formats may be implemented in two ways. The ADEI supports custom export plugins. Alternatively, it is
possible to filter exported data using system scripts. For example, to generate ROOT output, the data in CSV
format is piped to standard input of a simple ROOT application which converts it to ROOT format and prints to
standard output. The same mechanism could be used to compress output before returning it to the client application.
The chains of filters are supported. This allows to produce archived ROOT files. To limit amount of the exported data, 
a resampling by averanging or summing up can be requested. 

ADEI search engine is implemented with pluggable search modules and is able to search for channels,
channel groups, channel values, and time intervals. The channel search is very flexible. The channel names 
and descriptions are searched for single and multiple words, exact phrases, and regular expressions. The words are matched in 
three different ways: exact match, words starting with the search term, or words containing search term. The search is case-insensitive and all types of matching can be mixed in a single query. The value search finds a set of time intervals where the values of the given channel are above/below the specified threshold. The two modes are supported: search for time intervals where any value from the interval is above/below the specified threshold and search for time intervals where at least some of the values are above/below the threshold. The data cache is used to accelerate
searches over big amounts of data and all searches are executed within few hundred milliseconds. The interval search allows users to quickly position time axes. The search module supports strings like \emph{January 2005} or \emph{January - March, 2006}  and upon submitting of search request the time axis will be set accordingly.

Much of meteorological information is represented by time-series of multi-dimensional data such as cloud maps, etc. Though it is impossible to process dimmensional data using standard data aggregation chains in ADEI, we built a "Custom" data chain to provide a uniform access to  multi-dimensional data using standard interfaces and with minimal restrictions. The data source may optionally provide, so called Custom data, where each data channel is associated with custom array instead of scalar. The module handling the data from relational databases provides such Custom channels from the binary BLOB data stored in the database. As caching for multi-dimensional data is currently not available, the visualization modules requests data directly from the database. This limits applicability to rather small time intervals, but allows us to provide additional information in ADEI web interface while analyzing specific atmospheric events. As well, ADEI filtering subsystem is able to creatie derivative data channels based on the filter output. It allows us to feed multi-dimensional data into the chain of filters and extract scalar values characterizing some properties of the recorded data. For example, from the cloud map we can extract a cloud coverage in percent and make it available to the scientists as the standard ADEI time series.

\subsection{Fronted}
The main view of ADEI web frontend is represented on Figure~\ref{webfront} and the numeric labels from 
1 to 12 are used to reference interface elements in the description below. The main window (\emph{label~1}) 
contains plot depicting measurements of 7 sensors on a voltage, temperature, and default axes. The channel to axis
mapping can be defined either by the source database or in the ADEI configuration. All unmapped channels are 
displayed using optional default axes. There is no hard limit on a number of supported axes, but rather the size of the
browser window is only factor restricting the number of axes which can be reasonably displayed. 

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/webfront.eps}}
\caption{\label{webfront} Screenshot of ADEI Web Frontend. The data outage is indicated using a small line on top of the plot (see 5).
Legend contains description of displayed graphics. The selected part of plot may be zoomed or exported using buttons 3 and 4. 
Axes controls and results of search are located in the left sidebar. }
\end{figure}

The data for period of approximately two weeks is shown. 
During this period, the registrations of sensors were sampled into the database approximately ten times 
in a second what gives about 12 millions of data records over two week interval. For visualization,
this data is aggregated and approximately a few thousand data points are extracted from the caching 
database to render the graph. The system is optimized in a way that complete time of rendering does
not exceed a few hundred milliseconds on a standard desktop hardware. 
Sometimes, however, the collected information includes periods when no data was
recorded or existing recordings are invalid. Due to aggregation, the short outtages
is impossible to see on the low zoom levels. In order to handle such situations, ADEI 
includes a quality indication line on a top of the data plot (just below
a plot title). On the screenshot it is possible to see a tiny line indicating short period
when the data was not recorded due to power outtage (see \emph{label~5}).

The ADEI is configured using various controls in the sidebar. The lower part (\emph{label~7}) allows to set various 
options controlling behavior of export subsystem, data aggregation and visualization modes, etc.
The top part (\emph{label~6}) includes 3 tabs controlling which data is displayed on the plot. The \emph{Source} 
tab allow to select one of the pre-configured groups of channels. More flexibility can be 
achieved with \emph{Source Tree} tab of the bottom sidebar (\emph{label~7}) which allows
to select individual channels from the hierarchical tree. The \emph{Axes} tab allows to  tune 
the axes by specifying their ranges and switching between standard and logarithmic modes.
\emph{Time} tab is used to configure the time interval of interst. Few different modes are supported.
The data source may provide a list of time intervals when something important was happening. It is 
possible to select a desired interval from this list and apply it to the time axis. 
Alternatively, the beginning and the end of time axis could be set manually with a microsecond precision.
Other options include visualization of all stored data or just last quantity of seconds (i.e. plot for last minute,
hour, day, week, etc.). In the last case the plot will be periodically updated to display incoming data. 

It is also possible to zoom into the regions of interest on the plot using mouse. 
The subarea of plot can be selected using mouse pointer while holding left button (\emph{label~2}). After selection is 
made it still can be fine tuned: resized or positioned using mouse or keyboard arrows. 
The buttons in the right-bottom part are used to export data within selected time interval (\emph{label~3})
or to zoom into the selection (\emph{label~4}). Additional functional buttons can be implemented using
custom plugins. Also, the current plot on display can be zoomed in and out by scrolling mouse wheel. 
The default action is to zoom along time axis at the position of mouse pointer. However, the key modifiers may be used to
zoom over value axis or zoom in the center of the plot. The adjustments of plot position on the time and value axes are achievable
by scrolling mouse over the correspondent axis. The double click on considered axis will restore it into the automatic mode 
and the general overview will be displayed again.
Finally, the ADEI supports navigation history. \emph{Forward} and \emph{Back} buttons of the browser could
be used to go back and forth in the history. The URL in the navigation bar is always precisely
describing current position, selected time series, and all configured properties. This URL could be sent
to the colleagues over e-mail and exactly the same plot will be displayed on their PC.

A status bar (\emph{label~10}) is used to provide status and contextual messages to the user. Currently 
performed actions, their completion status, contextual help, emerging error messages are reported using 
status bar. On mouse movement the position of mouse pointer along all axes is reported as well.
An example could be seen on the provided screenshot. The color coding is used to help with association of axes.
ADEI also provides possibility to investigate graphics passing in the specified area of the plot. 
A legend window (\emph{label~10}) is popped up when the left mouse button is clicked. It contains a list of all graphics 
on the plot which are passing near position where mouse was clicked. The short name, description, and a range of values possessed in the neighborhood are presented on the legend.

ADEI provides advanced search and simple integrated WiKi engine. Upon entering a search string (\emph{label~11}), the bottom sidebar (\emph{label~7}) is opened and results are reported in the \emph{Search} tab. The example on screenshot displays results of searching for temperature sensors. The WiKi engine is normally used to describe the data channels available in the system and provides several specific extensions on top of standard WiKi syntax. \emph{[preview]} - generates a preview plot. The channel group, time interval, image size, aggregation mode, 
and other standard properties may be specified. The preview is linked and upon a click will switch to the plotting view
and display appropriate graph. \emph{[grouplist]} - generates linked previews for all channel groups available in the system.
\emph{[channels\_by\_name]} - includes alphabetical listing of all channels in the system. Upon a click the selected channel
will be plotted. \emph{[channels\_by\_group]} - includes hierarchical listing of all channels in the system. The selected channel will be
plotted upon a click as well. The example Wiki pages are depicted on Figure~\ref{adeiwiki}. 

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/adeiwiki2.eps}}
\caption{\label{adeiwiki} Screenshot of two ADEI Wiki pages. The first one lists the ASEC detectors and provides links on the pages with detailed detector description. The second page has previews of most interesting events. By clicking on the preview, the appropriate analysis session for the selected event will be opened in the main ADEI screen.}
\end{figure}

More inforamation in textual and visual forms may be provided in ADEI using so called secondary views. The secondary views are implemented with plugins which get all information about current ADEI screen on the display and all configured options. Based on thsese information, the relevant information is requested from ADEI backend and presented to the user in variety of forms. If necessary the views 
may interact with users by adding additional forms in the generated HTML content.

\section{ADEI Modules for Statistical Analysis}
A set of secondary modules is available for ADEI to extract statistical information from the time series and provide basic insight in cross-correlations between different channels. The particle detectors at ASEC measure modulation of the stable galactic cosmic ray “background” by local weather phenomenas like thunderstorms and additional flux of particles from the Sun.  Therefore, ADEI is often used to find a signal in the noisy background. A “Background” module is a core component of the analysis platform. Using ADEI standard display, it allows user to select an interval without any signal and extract the required statistics from it, see Figure~\ref{adeistat1}. For each of the data channels in ADEI, the module provides standard statistical information about the selected “background” to the users and other ADEI modules. To verify correctness, additionally the module shows a histogram and perform Gaussian fit.

Besides “Background”, there are several secondary ADEI modules which provide various statistical information about the data which is on display in the main ADEI window.  First of all, for each channel the “channel list” module reports minimum, maximum, average and standard deviation on the selected interval as well as the minimum and maximum in the units of standard deviation. Depending on the user selection, the standard deviation is either computed using the selected background or currently analyzed interval. For optimal performance, for larger time intervals an estimated value of standard deviation is computed using the averages from the ADEI cache.
The “histogram” module show the histogram of the selected channel currently on display, see Figure~\ref{adeistat1}. It allows to show the histogram of raw data or subtract the selected background first. The module supports normalization of the histogram and distribution and allows user to perform Gaussian fit of the displayed data. In the upper right corner of the plot user sees the mean value of variable in the selected data interval, the standard deviation and Pearson's chi-squared test value.  The number of bins are configured automatically or may be specified by the user if required. Again for optimal performance, if larger time interval is selected, the histogram is estimated based on the averages in the ADEI cache. 

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/adeistat1.eps}}
\caption{\label{adeistat1} In the left part of the figure, a “background” selection module is illustrated. In the main ADEI screen, the user selects the time interval with only background noise present and the statistical information about the selected interval is shown on the secondary ADEI screen. The right part of the figure presents “histogram” module. Count rates of the 3 cm thick outdoor plastic scintillator are shown in the main ADEI screen and corresponding histogram along with statistical information is visualized in the secondary screen.}
\end{figure}

The “scatter plot” module is used to visualizes relation of 2 channels, see Figure~\ref{adeistat2}. In the upper right corner of the scatter plot the linear correlation coefficient of the selected pair of time series is depicted. Delays between signals registered by different detectors can be found out using the “correlation plot” module. It allows to view the dependence of the correlation coefficient depending on the added delay of one of time series related to the second one. The user is expected to specify a step in the seconds, minutes, hours, or days.

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/adeistat2.eps}}
\caption{\label{adeistat2} The left plot demonstrates “scatter plot” functionality and the “correlation plot” is depicted in the right part of the figure. During TGE (Thunderstorm Ground Enhacement) one and the same thunderclouds send particles to the Earth surface and the time series of different spatially distant detectors are highly correlated as it can be seen in the left part. In the right part, we check the relation between atmospheric pressure and count rate of the ASEC detectors. As there is no delay between atmospheric pressure and count rate, we conclude that the pressure decreases the same minute the count rate increases.}
\end{figure}

Traditionally, the particle detectors were counting a number of pulses per unit of time. The modern electronics allows to measure parameters of detected pulse (amplitude, width, shape or total charge of pulse) as well. In High Energy Physics, usually, parameters of measured pulse proportional to registered particle energy. By measuring parameters of pulse we can obtain particle energy spectrum. A new visualize module for spectrum data was developed.  The spectrums are represented as multichannel time series where each channel corresponds to a given energy interval. The spectrums are stored in MySQL database and each record represents spectrum measured at a given time. Such organization allows us to use standard ADEI capabilities to access and cache spectrum data. We also analyze spectrums as they recorded and generate additional time series representing mean, median, and standard deviation of the recorded spectrums. Theses time series are displayed using ADEI as well. 
Into the ADEI data analysis platform a “spectrum” module have been added. In the configuration, the associations between ADEI channels and spectrums are defined. The “spectrum” module allows to select one of the spectrums associated with the channels currently displayed in the ADEI main screen and shows it using standard bar plot, see Figure~\ref{adeispect}. Additional statistical information is shown bellow. The spectrum visualization subsystem is integrated with “background” module and allows to display the raw spectrums or to subtract the spectrum of the background from the measured values first. Since the visualization of large time intervals can be requested, the module supports visualization of the aggregated data. In this case minimum, average, and maximum values are displayed.  Alternatively the sum over the selected time interval can be requested using Draw Mode control. In all supported modes, the module allows to fit displayed data using  exponential, logarithmic, a and power functions.

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/adeispect.eps}}
\caption{\label{adeispect} Particle count rates (ASNT 60cm #1) are displayed on the main chart and corresponding spectrum data in the sidebar on the left.}
\end{figure}

\section{Discussion}
The ADEI platform delivers following statistical analysis capabilities within an intuitive modeling interface that can be easily understood by users:
\begin{enumerate}
\item Gaussian analysis of selected time series with fit estimation;
\item Scatter plots and delayed correlation analysis;
\item Calculation of the P-values for the background only hypothesis in terms of one-sided Gaussian tail significances and provided in units of standard deviation;
\item Resolving the mixture of distributions, recovering of the fractions of different classes;
\ite Presenting of time series in percent and standard deviations;
\item Combination of time series.  
\end{enumerate}

In this section we will discuss how the provided options are used for estimation of important physical parameters and for the paper and report composing. As one can see in Figure~\ref{tgerain}, the all TGEs (Thunderstorm ground enhancement event) observed in June 2013 was accompanied by rain. Rain started during TGE in progress and after it stops TGE fast declines. The TGE amplitude is approximately proportional to the rain rate. Consequently, we can deduce that charge is resided on the rain droplets. The positive and negative ions can be separated in the droplet under the action of the ambient electric field, thus forming two residual stretched charged clusters. Therefore, the upper part of droplet forms with main negative layer of the thundercloud the lower dipole accelerated electrons downward; and the negatively charged bottom of the droplet is responsible for the large negative near surface electric field measured by the EFM-100 electrical mill. The TGE amplitude should be proportional to the total positive charge in LPCR; and, therefore - to the amount of rain droplets (water) in the bottom of cloud. An estimate of amount of water in cloud is the rain rate. For the TGEs on June 20-21 (right side of Figure~\ref{tgerain}) the charge accumulated in the droplets was not sufficient to provide strong electric field to unleash RREA process and we detect only modest enhancements of particle fluxes due to MOS process. On June 16-19 the rain rate was sufficient to stipulate large and prolonged TGEs.

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/tgerain.eps}}
\caption{\label{tgerain} Time series of the rain rate (bottom); time series of the count rate of outdoor plastic scintillator with energy threshold 1.5 MeV(middle); time series of the disturbances of near surface electric field (top).}
\end{figure}

Zooming over the time axis we can investigate each TGE in more details, see Figure~\ref{sigmaplot}. If we post standard count rate values due to very large count rate of 5 cm thick scintillators of ASNT, the count enhancement of 30 MeV electrons will not be visible; with standard deviation we can see that significance of the larger than 30 MeV electrons is rather large, above 6 sigma.

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/sigmaplot.eps}}
\caption{\label{sigmaplot} Largest TGE ever detected at Aragats; the amplitude of TGE is denote in registered particles (left) or in numbers of standard deviations (right)}
\end{figure}

Gaussian fit for time series gives direct evidence on the existence of the TGE or another astrophysical source which disordered time series (also existence of outliers in the histogram can signal on a technical disruption and should be notified for immediate repairs). The monitoring of huge number of the measuring channels at Aragats Space Environmental center (ASEC) requires automation of peak searching procedures and detecting of outliers (greater than 4 sigma). The visualization of outliers, the Gaussian normalization and overlaying standard Gaussian plot on raw data histogram is now available in ADEI, see Figure~\ref{adeifits}.

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/adeifits.eps}}
\caption{\label{adeifits} The TGE detected 12 MAy 2013 (right) and the time series before start of TGE (left)}
\end{figure}

We illustrate a complete ADEI workflow implementing visualization/analysis methods to the largest TGE of Spring 2014. At June 10 - 11 several thunderstorms occurred at Aragats, see Figure~\ref{tge20140610}. One of them was accompanied by the abrupt increase o cosmic ray flux at 17:31 on June 11. The successive steps of the “semi on-line” analysis of this event are depicted in Figure~\ref{vmavdemo} and seven inserts to it. 
First of all images of multivariate time series are available on personal and public screens at CRD. Physicists are routinely monitoring the severe storms, lightning occurrences, meteorological parameters and, of course, time series of particle fluxes. The automatic monitoring programs operating in CRD site as well report on severe weather conditions and enhanced particle fluxes. When, as it is seen in Figure~\ref{vmavdemo}, we detect abrupt enhancement of the particle flux the following stages of statistical and physical analysis are started.
\begin{enumerate}[label*=\arabic*.]
\item Prove that TGE is not “fake” event due to failure of equipment: 
\begin{enumerate}[label*=\arabic*.]
\item Check electric field disturbances – usually TGE happens on high negative near surface electric field.
\item Check of air humidity – usually relative humidity exceeds 90% when TGE started.
\item Observe 1-second time series to look in details of particle flux enhancement and discover non-trivial structures in the time-series pattern – the electronics failures or nearby lightning interferences have specific shapes easily to distinguish from TGEs.
\item Examine remote detectors with independent electronics and power supplies.
\item Look in low energy and high-energy particle fluxes (among numerous particle detectors at Aragats there are ones with varying energy threshold). If low and high-energy particle fluxes registered on one and the same detector with one and the same electronics are different (low energy demonstrate enhancement and high – not) it is surely not lightning interference.
\end{enumerate}
\item Enumeration of peaks and generation of figures
\begin{enumerate}[label*=\arabic*.]
\item In the ADEI screen measure of the peaks heights. In Figure~\ref{vmavdemo} we can see that the height of 100 combination of STAND1 detector is 39,600 particles per minute and for 010 combination – 26,500 particles per minute.
\item Calculate mean and variance of time series before the TGE event, see inserts a) and b) to Figure~\ref{vmavdemo}.
\item Calculate difference of peak and mean values and then using variance estimate the peak significance using the P-values for the background only hypothesis.
\item Make plots of peak in % of background and in P-values, see inserts e) and f) to Figure~\ref{vmavdemo}.
\item If necessary, join minutes of time series – from 1 to 3 minutes, or more. On insert g) of Figure~\ref{vmavdemo} we can see that 3-minute time series demonstrate larger significance comparing with 1-minute (~140 comparing with ~90);
\end{enumerate}
\item Make correlation analysis of 2 time-series. In inserts c) and d) we see that both time series exactly correlated without any time lag. Time lag can arise from error in synchronization of different detectors or due to delay of different particle fluxes. 
\item Recover electron and gamma ray fluxes incident on detector. By the detector response calculation and calibration with cosmic ray flux we know efficiencies of each scintillator of the STAND1 detector to register electron and gamma ray. Knowing these efficiencies by solving the system of linear equations it is possible to resolve mixture of particle fluxes and estimate separately fluxes of electrons and gamma rays. As we can see in Figure~\ref{vmavdemo} flux of gamma rays at 10 June was very large reaching approximately 5000 gamma rays per second per m.sq. Flux of electrons is only 3.7% of gamma ray flux due to fast attenuation of charged flux in atmosphere. Another option of the ADEI statistical package (Integral option) allows to calculated total number of TGE particles; at 10 June the total number of particles was 65.750.
\end{enumerate}

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/tge20140610.eps}}
\caption{\label{tge20140610} Illustration of the physical analysis of the TGE event with ADEI platform}
\end{figure}

\begin{figure}
\makebox[\textwidth]{\includegraphics[width=\textwidth]{img/vmavdemo.eps}}
\caption{\label{vmavdemo} Illustration of the physical analysis of the TGE event with ADEI platform}
\end{figure}


\section{Conclusion}
With the growing archives of the time series from the monitoring of the various cosmic ray fluxes and atmospheric electro-magnetic field at ASEC, the need to establish a new type of infrastructures for using and comparing the data from numerous sources becomes more and more urgent – ADEI with new statistical modules meets this need. To acquire the expected new knowledge, joint data samples from different domains are joining to make multivariate correlation analysis. The developed methodology links a multitude of space and geophysical observations into an integrated system that provides analysis tools and services to fully utilize the scientific potential of current and future space weather/geophysical observations. Physicists of Cosmic ray division prepare and publish in high-rank scientific journals near 20 articles heavily using first ADEI platform in 2012 - 2015.  ADEI allows performing research projects very fast and comprehensive. We send the paper to Journal of Geophysics research discussing very interesting events observed on Aragats in April on May 3. And paper included event from April 28. ADEI tools make analytical work on very sophisticated problems rather easy; one can try and test many hypotheses very fast and come to correct conclusion.     


\newpage 
\section*{References}
\bibliography{adei}

\end{document}