Overview

HDP (Hyper Data Platform) provides enterprises with unified data access, synchronization and processing capabilities, and supports the construction of visual data synchronization processes between different databases, message queues and worksheets. The platform provides efficient and stable data flow capabilities for business systems through three types of task modes: manual, scheduled and real-time, and realizes multi-source data integration and processing without coding. HDP private deployment version is a container deployment version of HDP SaaS service, which allows enterprise users to establish data synchronization capabilities on their own cloud computing facilities.

Design plan

HDP is based on a microservice architecture and is deployed using Docker images. In private deployment mode, in order to ensure system stability while reducing deployment difficulty, we modified this more complex microservice dependency relationship and finally adopted a single image mode based on microservice architecture to merge independent microservice images into one large image (microservice application image). The configuration file is built-in and supports expansion. Data storage component images (including: MySQL, MongoDB, Redis, Kafka, MinIO) are also provided, which are enabled by default in standalone deployment mode. In addition, file object storage based on MinIO is also used, and finally a command is implemented to start the HDP service.

The standalone deployment mode is inseparable from the problem of single points of failure, but the single image of the HDP private deployment version is more to simplify deployment. It is still a collection of microservices (similar to the multi-container Pod in Kubernetes). Therefore, in order to ensure the availability of each service process in the container, a health check thread is preset inside the container, and it can automatically recover when a service fails. View [standalone deployment mode](./deployment/source.md#standalone deployment mode)

The cluster deployment mode is a more fine-grained clustering of each microservice and dependent component, mainly including storage component (such as: MySQL, MongoDB, Redis, Kafka, etc.) clusters and microservice application clusters. The microservice application image itself has adapted to the standalone and cluster deployment modes. View [cluster deployment mode](./deployment/source.md#cluster deployment mode)

Architectural features

Microservice architecture and containerized deployment

HDP adopts a microservice architecture to split system functions into multiple independent and scalable service units, allowing each module to run autonomously. This design enhances the system's response speed, fault tolerance, and maintenance efficiency, and supports agile development for cross-team collaboration. Each microservice can be managed and scheduled independently to achieve functional isolation and efficient invocation.

Combining Docker and Kubernetes containerization technologies, HDP's microservice components can run efficiently across environments, greatly simplifying system deployment and management. Containerization abstracts the underlying hardware resources and uses Kubernetes' elastic scaling, automatic expansion, and load balancing capabilities to enable the system to remain stable in response to sudden business demands. In addition, HDP integrates middleware such as cache and message queue, which further improves system performance and stability in high concurrency scenarios.

Multi-level deployment model

The HDP private deployment version provides flexible deployment environment support, allowing users to choose suitable cloud computing resources based on their own resources. Whether it is a virtual machine, a physical machine or a cloud host, HDP is compatible. With such diverse deployment options, HDP can better adapt to the IT budgets and infrastructure conditions of different enterprises.

In order to adapt to the resource investment of enterprises of different sizes, HDP supports standalone and cluster deployment modes. Standalone deployment is suitable for small and medium-sized enterprises or test environments, while cluster mode provides flexible resource options for larger enterprises.

In addition, HDP's cluster architecture supports dynamic resource adjustment and mode switching. As business develops or expands, users can seamlessly upgrade from streamlined configuration to higher specifications, or choose to appropriately reduce configuration when resource utilization decreases to ensure optimization of resource investment.

Service instance optimization and scene adaptation

HDP provides service instance management functions based on actual business needs, allowing enterprises to adjust the number of instances as needed to better adapt to different scenarios. Through preconfigured templates, enterprises can flexibly choose configuration strategies based on business loads, such as configuring more copies for frequently accessed services, reducing resource consumption of non-core modules, and thus optimizing overall costs.

This design improves the scalability and resource utilization of the system, ensuring that HDP can quickly respond to new functional requirements and adapt to diverse application scenarios. The independent scalability of service modules allows HDP to iterate quickly when developing and integrating new features, helping enterprises maintain technology advantages and cope with business fluctuations.

Visual installation experience

In order to help users get started and experience the product faster, the HDP private deployment version provides a free professional version trial. Users can complete product installation and configuration through simple visual guided steps without the support of a professional IT team. Through a series of intuitive interfaces and detailed installation instructions, the user's learning curve is greatly reduced and a convenient initial experience is provided.

Comprehensive operations management and system observability

HDP is equipped with a powerful operations manager, which provides system monitoring, alerting, backup and recovery support from many aspects to ensure the observability and robustness of the system. Includes:

Monitoring: Real-time monitoring of system resource usage, including key indicators such as CPU, memory, and network traffic. Through graphical display, operations personnel can promptly discover and deal with performance bottlenecks.

Alerts: Supports multi-level alarm rule configuration, and notifies key events through email, SMS or third-party tools to facilitate quick response by operations personnel.

Data Management: Provides data archiving and backup functions to ensure the durability and security of key data. It also supports dynamic adjustment of multiple storage architectures to meet business expansion needs.

Performance Monitoring: Carry out independent performance monitoring and optimization suggestions for the core functions of the system to ensure the process operation of the system.

Iteration cycle

The functions of the HDP private deployment version and the HDP SaaS version are basically the same. The iteration speed of the SaaS version is faster, and the synchronization of functions to the private deployment version is generally about one month apart. This is mainly because version merging, testing, and defect repair of new functions require a certain period of time. The purpose is also to ensure the stability of the new private deployment version.