Spark 3.5 LTS 部署实战：Standalone/YARN 集群 3 节点配置与端口解析

Spark 3.5 LTS 集群部署实战Standalone与YARN模式深度配置指南1. 环境准备与基础架构设计在开始部署Spark 3.5 LTS集群前需要明确硬件资源配置与网络拓扑。典型的生产环境采用3节点架构1个Master节点和2个Worker节点每个节点建议配置16核CPU、64GB内存和1TB SSD存储。这种配置能够平衡成本与性能满足中等规模数据处理需求。系统依赖检查清单JDK 8/11推荐Amazon CorrettoScala 2.12.xPython 3.8如需PySpark支持SSH无密码登录配置统一的NTP时间同步# 基础环境验证命令 java -version scala -version python3 --version ssh master01 hostname # 测试SSH连通性网络方面需要确保以下端口互通Master节点7077RPC、8080Web UIWorker节点随机端口范围默认4040-4149历史服务器180802. Standalone模式部署详解Standalone模式是Spark内置的集群管理器适合快速搭建测试环境或中小规模生产部署。以下是关键配置参数conf/spark-env.sh# 核心参数配置示例 SPARK_MASTER_HOSTmaster01 SPARK_MASTER_PORT7077 SPARK_WORKER_CORES16 SPARK_WORKER_MEMORY56g # 保留8GB给系统 SPARK_WORKER_INSTANCES1 SPARK_DAEMON_MEMORY4g安全增强配置# spark-defaults.conf spark.authenticatetrue spark.authenticate.secretyour_complex_secret spark.acls.enabletrue spark.ui.filtersorg.apache.spark.deploy.master.AccessFilter启动集群的推荐方式是通过systemd服务管理# /etc/systemd/system/spark-master.service [Unit] DescriptionApache Spark Master Afternetwork.target [Service] Typeforking ExecStart/opt/spark/sbin/start-master.sh ExecStop/opt/spark/sbin/stop-master.sh Userspark节点健康检查脚本#!/usr/bin/env python3 import requests from datetime import datetime def check_spark_node(host, port8080): try: res requests.get(fhttp://{host}:{port}/json/, timeout5) data res.json() return { alive: True, workers: data.get(workers), memory: data.get(memory), status: data.get(status) } except Exception as e: return {alive: False, error: str(e)} if __name__ __main__: nodes [master01, worker01, worker02] print(fCluster health check at {datetime.now():%Y-%m-%d %H:%M}) for node in nodes: status check_spark_node(node) print(f{node}: {OK if status[alive] else DOWN} - {status})3. YARN模式集成方案对于已有Hadoop环境的企业YARN模式能更好地利用现有资源。需要特别注意资源分配的竞争策略关键YARN配置!-- yarn-site.xml -- property nameyarn.nodemanager.resource.memory-mb/name value57344/value !-- 56GB -- /property property nameyarn.scheduler.maximum-allocation-mb/name value57344/value /property property nameyarn.nodemanager.resource.cpu-vcores/name value16/value /propertySpark侧需要调整的对应参数# spark-defaults.conf spark.masteryarn spark.submit.deployModecluster spark.driver.memory8g spark.executor.memory12g spark.executor.cores4 spark.executor.instances8 spark.yarn.executor.memoryOverhead2g动态资源分配配置spark.dynamicAllocation.enabledtrue spark.dynamicAllocation.initialExecutors2 spark.dynamicAllocation.minExecutors2 spark.dynamicAllocation.maxExecutors20 spark.shuffle.service.enabledtrue4. 核心服务端口解析与安全加固Spark集群涉及多个关键服务端口需要明确其作用并实施安全防护端口号服务类型默认绑定安全建议7077Master RPC0.0.0.0配置IP白名单8080Master Web UI0.0.0.0启用HTTPSBasic Auth4040App Web UIDriver节点动态端口范围访问控制18080History Server独立服务器与HDFS ACL集成8081Worker Web UIWorker节点限制内网访问Nginx反向代理配置示例提升Web UI安全性server { listen 443 ssl; server_name spark-ui.example.com; ssl_certificate /path/to/cert.pem; ssl_certificate_key /path/to/key.pem; location / { proxy_pass http://master01:8080; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; auth_basic Spark Cluster; auth_basic_user_file /etc/nginx/.spark_htpasswd; # 限制访问IP allow 192.168.1.0/24; deny all; } }审计日志配置log4j.propertieslog4j.logger.org.apache.spark.deploy.master.AuditLoggerINFO, audit log4j.appender.auditorg.apache.log4j.DailyRollingFileAppender log4j.appender.audit.File/var/log/spark/audit.log log4j.appender.audit.DatePattern.yyyy-MM-dd log4j.appender.audit.layoutorg.apache.log4j.PatternLayout log4j.appender.audit.layout.ConversionPattern%d{yyyy-MM-dd HH:mm:ss} %p %c{1}: %m%n5. 自动化部署与监控方案使用Ansible实现集群的自动化部署# playbook.yaml - hosts: spark_cluster vars: spark_version: 3.5.0 install_dir: /opt/spark tasks: - name: Install Java apt: name: openjdk-11-jdk state: present - name: Download Spark get_url: url: https://archive.apache.org/dist/spark/spark-{{ spark_version }}/spark-{{ spark_version }}-bin-hadoop3.tgz dest: /tmp/spark.tgz - name: Extract Spark unarchive: src: /tmp/spark.tgz dest: {{ install_dir }} remote_src: yes extra_opts: --strip-components1 - name: Configure environment template: src: templates/spark-env.sh.j2 dest: {{ install_dir }}/conf/spark-env.sh mode: 0644 - name: Start Master (on master node) systemd: name: spark-master enabled: yes state: started when: inventory_hostname master01监控指标采集Prometheus配置scrape_configs: - job_name: spark metrics_path: /metrics static_configs: - targets: [master01:8080, worker01:8081, worker02:8081] - job_name: spark_jmx static_configs: - targets: [master01:9999, worker01:9999, worker02:9999] relabel_configs: - source_labels: [__address__] target_label: instance regex: (.*):\d replacement: ${1}关键监控指标告警规则示例groups: - name: spark-alerts rules: - alert: SparkExecutorFailed expr: spark_executor_status{statusFAILED} 0 for: 5m labels: severity: critical annotations: summary: Spark executor failed on {{ $labels.instance }} - alert: HighDriverMemoryUsage expr: (spark_driver_jvm_memory_used_bytes / spark_driver_jvm_memory_max_bytes) 0.9 for: 10m labels: severity: warning6. 性能调优实战技巧内存优化策略Executor堆外内存配置公式spark.executor.memoryOverhead max(384MB, 0.1 * spark.executor.memory)序列化优化组合spark.serializerorg.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.max512m spark.kryo.registrationRequiredtrue动态分区优化-- 在Spark SQL中启用动态分区 SET spark.sql.sources.partitionOverwriteModedynamic; SET hive.exec.dynamic.partitiontrue; SET hive.exec.dynamic.partition.modenonstrict;Join优化对照表策略适用场景配置参数Broadcast Join小表(10MB)关联大表spark.sql.autoBroadcastJoinThresholdSort-Merge Join大表关联已分区排序spark.sql.join.preferSortMergeJoinBucket Join预分桶表关联spark.sql.sources.bucketing.enabledShuffle Hash Join中等表关联内存充足spark.sql.join.preferSortMergeJoinfalse自适应查询执行(AQE)配置spark.sql.adaptive.enabledtrue spark.sql.adaptive.coalescePartitions.enabledtrue spark.sql.adaptive.advisoryPartitionSizeInBytes128MB spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin0.27. 故障诊断与日常维护常见问题排查流程检查Master/Worker日志tail -100f /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-*.out验证网络连通性for node in master01 worker01 worker02; do echo Checking $node: nc -zv $node 7077 nc -zv $node 8080 done资源冲突检查ps aux | grep -i spark | grep -v grep netstat -tulnp | grep -E 7077|8080|4040关键维护操作滚动升级步骤先升级Worker节点最后升级Master节点使用--kill优雅终止运行中的应用数据清理策略# 清理事件日志 find /spark-event-logs -type f -mtime 30 -delete # 清理临时目录 hdfs dfs -rm -r /tmp/spark-*备份恢复方案import datetime import subprocess def backup_spark_conf(): timestamp datetime.datetime.now().strftime(%Y%m%d_%H%M) backup_dir f/backup/spark_conf_{timestamp} subprocess.run([mkdir, -p, backup_dir]) subprocess.run([cp, -r, /opt/spark/conf, backup_dir]) subprocess.run([tar, -czf, f{backup_dir}.tgz, backup_dir]) subprocess.run([hdfs, dfs, -put, f{backup_dir}.tgz, /backups/spark_config/]) print(fBackup completed: {backup_dir}.tgz)